Testing Custom Hooks and Complex Interactions

Not every important behavior manifests as a visible widget. Hooks consolidate data fetching, subscriptions, optimistic updates, and cross-cutting concerns like analytics or feature flags. Testing them well means choosing the right level of isolation: sometimes a hook is the unit, sometimes the hook is an implementation detail and a parent component test is clearer, and sometimes the only honest check is a browser-level interaction. Advanced React codebases use all three.

renderHook for hook-focused behavior

React Testing Library’s renderHook mounts a minimal component that calls your hook and exposes the latest result. It is the right tool when the hook’s contract is public API—shared across screens—or when logic is complex enough that component tests would bury the signal in setup noise.

import { renderHook, act } from "@testing-library/react";
import { useCounter } from "./useCounter";

describe("useCounter", () => {
  it("initializes with provided value", () => {
    const { result } = renderHook(() => useCounter(5));
    expect(result.current.count).toBe(5);
  });

  it("increments count", () => {
    const { result } = renderHook(() => useCounter());
    act(() => result.current.increment());
    expect(result.current.count).toBe(1);
  });
});

Wrap mutations that trigger state updates in act when you bypass RTL’s higher-level helpers so React flushes updates before you assert.

Providers and composition

Hooks that read context cannot run in a vacuum. Supply a wrapper that mirrors the production provider stack—often trimmed to the smallest subtree that satisfies dependencies:

function Wrapper({ children }: { children: React.ReactNode }) {
  return <AuthProvider value={mockAuth}>{children}</AuthProvider>
}

const { result } = renderHook(() => useAuthUser(), { wrapper: Wrapper })

The judgment call is how faithful the wrapper must be. A stub context value is fine when testing pure consumers. When integration bugs tend to come from provider ordering or default values, prefer a wrapper closer to the real app shell for a smaller number of high-value tests rather than duplicating the entire tree everywhere.

Async hooks and data fetching

Async hooks usually move through loading → success or error. Tests should assert that sequence, not only the happy ending:

import { renderHook, waitFor } from "@testing-library/react";

it("fetches data on mount", async () => {
  const { result } = renderHook(() => usePosts());

  expect(result.current.isLoading).toBe(true);

  await waitFor(() => {
    expect(result.current.isLoading).toBe(false);
  });

  expect(result.current.posts).toHaveLength(2);
});

Pair this style with MSW so the fetch path is realistic. If you mock modules instead, keep the mock’s shape aligned with MSW handlers or generated types; divergent doubles are a frequent source of “tests pass, app fails.”

Exercise error and empty branches with the same care as success: override handlers to return 401, 422 with validation JSON, or malformed payloads when your hook normalizes failures. Hooks that retry, back off, or cancel in-flight work should prove that updates do not leak after unmount or dependency change—often by resolving a promise after teardown and asserting the UI and hook state stay stable.

Global stores: reset real state instead of mocking everything

For Zustand, Jotai, or similar libraries, advanced teams often integrate the real store in tests and reset state between cases. That exercises selectors, middleware, and persistence guards together:

import { useUserStore } from '@/stores/user'

beforeEach(() => {
  useUserStore.setState({ user: null })
})

it('logs out user', async () => {
  useUserStore.setState({ user: mockUser })
  const user = userEvent.setup()
  render(<Header />)

  await user.click(screen.getByRole('button', { name: /logout/i }))

  expect(useUserStore.getState().user).toBeNull()
})

The anti-pattern is mocking the store module in every file until no behavior remains. The middle path is mocking side boundaries—HTTP, localStorage, clocks—while letting state management run for real.

Jotai and other atomic models follow the same reset discipline: recreate stores or clear atoms in beforeEach so derived state does not leak between cases. When hooks subscribe to ResizeObserver, matchMedia, or WebSocket feeds, wrap those behind small adapters you can fake in tests while still exercising the hook’s orchestration logic.

React 19: Server Actions and useActionState

When forms call Server Actions, tests typically mock at the module boundary where the action is imported, because the server runtime is not available in jsdom. Prefer verifying observable outcomes: pending UI, validation messages, optimistic transitions, and the arguments passed to a mocked action. For useActionState, render the form, submit it with userEvent, and assert how state evolves; use waitFor when intermediate optimistic states matter.

vi.mock("./actions", () => ({
  submitProfile: vi.fn(),
}));

Keep assertions user-facing where possible—error text, disabled controls—so the test remains stable if you refactor how state is stored internally.

Vitest Browser Mode for what jsdom cannot see

jsdom is not a browser. Layout, CSS containment, scroll semantics, and some input events differ enough that regressions slip through. Vitest’s browser mode runs tests in a real engine—commonly Chromium via Playwright—so component tests can catch issues that only appear with true rendering:

export default defineConfig({
  test: {
    browser: {
      enabled: true,
      provider: "playwright",
      instances: [{ browser: "chromium" }],
    },
  },
});

Use browser mode selectively. It is slower and heavier than jsdom; reserve it for components where geometry, focus rings, portal positioning, or native controls are central to correctness. The rest of the suite stays fast; a thin layer of browser tests guards the places where fidelity matters.

Drag-and-drop, multi-step wizards, and nested portals often sit at the boundary between RTL and Playwright. If assertions depend on layout pixels or browser-only APIs, favor browser mode or E2E; if they are about ARIA state and visible text, jsdom is usually enough.

Putting it together

Hooks and complex interactions reward a simple decision tree: if users depend on it, assert through the UI; if collaborators depend on it as an API, exercise it with renderHook; if the browser’s native behavior is the risk surface, promote a focused test to Vitest’s browser mode. That layering keeps advanced React applications testable without turning every file into a simulation of the entire platform.