Reference

Organization Design
for Complex Worlds

How does horizontal complexity shape organization design?

This interactive gives a visual introduction to the central mechanism in Organization Design for Complex Worlds. It is intended for economists and other readers who are comfortable with technical economic models. The paper uses spectral theory to study how organizations should design themselves, but no prior background on these topics is assumed here. If you are comfortable with variance and conditional expectation, you have everything you need.

The full interactive takes about 15 minutes to explore, including some time with the simulations. It may take somewhat longer for readers who choose to explore the optional technical details. By the end, you will see how attention is allocated in a complex environment and why the way uncertainty is distributed—not only its total amount—shapes team size. The interactive omits proofs and several extensions; see the paper for the complete analysis.

The central idea is that the organizational effect of complexity depends on where uncertainty lies relative to managerial attention.

Use Reference at the bottom left for notation, and Sections at the bottom right to jump between parts.

The Organization's Problem

The main design question studied in the paper is how large to make workers' teams.

Workers can adapt more efficiently to local conditions in smaller teams.

However, larger teams help headquarters coordinate more efficiently, because coordination activities are at the team rather than individual level.

To formalize this tradeoff, think of the organization's work as a continuum of tasks indexed by x ∈ [0,1]. Each task has a locally relevant state W0(x), and workers are grouped into teams of size k. The paper assumes that W0(x) is a Gaussian process. We will see what that looks like—and why it is useful—very shortly.

The organization chooses team size k ≥ 1 to minimize an objective containing these two kinds of loss:

The first term captures coordination loss, while the second captures adaptation loss.

Adaptation loss captures the cost of preventing workers from tailoring their actions fully to local conditions. It rises with team size through f(k). Coordination loss captures headquarters' inability to predict the actions taken within each team. Larger teams reduce this loss by reducing the mass of team-level states headquarters must track.

Here, S is a signal process chosen by headquarters subject to a limit on how informative it can be. Section 4 shows how headquarters chooses this signal and what coordination loss remains.

More on the model

More on the model. A worker at location x chooses an action ax to minimize

f(k)[ax² + (W0(x) − ax)²].

The optimal action is

ax = W0(x)/2,

which yields per-worker adaptation loss

f(k)W0(x)²/2.

Appendix A motivates the team-state process using a discrete-location model. Workers are grouped into contiguous teams of size k, and each team's state is the average action of its members. As the number of workers grows while team size remains fixed, the limiting team-state process is

W1(u) = ½W0(ku),  u ∈ [0, 1/k].

In particular, the mass of teams is 1/k times the mass of workers.

Headquarters cannot observe these team states directly. Instead, it chooses a signal process S, subject to a mutual-information bound described in Section 4. Coordination loss depends on the squared distance between headquarters' posterior expectation of each team state and the true team state.

Section 3 of the paper presents the full model, while Appendix A explains in more detail how the continuum formulation approximates a many-location model.

The interactive walks through the three steps behind the solution method from the paper:

  1. How can the correlated task-state process be decomposed so that coordination and adaptation can be expressed in the same terms?
  2. What does headquarters learn about, and how much coordination loss remains?
  3. How do those losses change as team size changes?

Decomposing the Task-State Process

The first step is to rewrite the task-state process in a way that will enable a clean comparison between coordination and adaptation losses. The strategy is to decompose this process into a sequence of independent parts.

This is one realization of the task-state process W0(x): a sample path showing how the state varies across task locations.

It is assumed to be a mean-zero Gaussian process with covariance kernel C(x,x′). This means that the states at any finite collection of locations are jointly normally distributed, with C(x,x′) giving the covariance between the states at x and x′.

The covariance kernel records how strongly the states of two tasks move together. Nearby tasks may call for similar actions, while distant tasks may behave differently.

Because W0(x) has mean zero,

This integral measures the total amount of variation across the task landscape. It will determine adaptation loss—but, importantly, total variation alone will not determine team size.

The decomposition of W0(x) into independent components will suggest another way of writing adaptation loss, but more important is that this makes the attention problem tractable. Intuitively, this step is necessary because attention is global: headquarters must decide what to learn about the entire correlated process at once.

Decomposing W0 into independent components

Why the Decomposition Helps

The animation you just saw illustrates the Karhunen–Loève decomposition:

Each term is an independent mode. The function φj(x) describes a pattern of variation across tasks, Zj determines how strongly that pattern appears in this particular realization, and λj measures how much variance the mode contributes.

We order the modes from the largest contribution to the smallest:

λ1 ≥ λ2 ≥ ….

The decomposition implies

This is the key reduction: the same values that describe how variation is distributed across the environment will also determine where headquarters directs its attention.

Formal definitions and deriving this identity

Formally, the functions φj are the eigenfunctions and the values λj are the corresponding eigenvalues of the covariance operator generated by C(x,x′). An eigenvalue–eigenfunction pair (λj, φj) satisfies

∫01 C(x,x′)φj(x′)dx′ = λjφj(x).

The eigenfunctions are orthonormal:

∫01 φj(x)² dx = 1

and, for j ≠ ℓ,

∫01 φj(x)φℓ(x) dx = 0.

Substituting the Karhunen–Loève representation into E[∫01W0(x)² dx] and using orthonormality makes the cross-terms vanish. Each remaining mode contributes λj, giving

E[∫01W0(x)² dx] = ∑λj.

Since C(x,x) = E[W0(x)²], this also implies

This is the continuous-operator analog of the matrix identity that the trace equals the sum of the eigenvalues.

If you have seen principal-components analysis before, the Karhunen–Loève decomposition is essentially its continuous-process analog. With finitely many task locations, the same exercise would be ordinary principal-components analysis of a covariance matrix.

The ordered collection of eigenvalues is sometimes called the spectrum of the covariance operator.

The organization's problem can therefore be written as:

Allocating Headquarters' Attention

The decomposition lets us describe headquarters' attention problem more precisely.

Headquarters can design a signal informative of the team-state process, but it does so with a fixed attention budget ρ. Its goal is to use that budget in the way that reduces prediction error about the team-state process the most.

Formal definition of information

Mutual information measures how informative a signal is about an unknown state. The specific proposal in the paper is to measure how informative S is about the team-state process W1 by considering every finite collection of locations:

Headquarters may choose any signal process satisfying

I(W1;S) ≤ ρ.

The parameter ρ is therefore a bound on the total information headquarters can acquire about the process.

To see how the solution works, hold team size fixed at k = 1 for now. Later we will see what changes when k can vary.

For multivariate normal states, the solution is classical and is known as water-filling. Attention should be spent where it benefits headquarters the most, which depends on where most of the variation lies. This is exactly where the decomposition is useful: it separates total variation into distinct dimensions, allowing headquarters to reduce uncertainty along each dimension separately.

Here is how it works

Headquarters allocates attention to the modes with the largest λjs and equalizes posterior variance across all modes that receive attention.

Let μ* denote this common posterior variance, called the water level.

If λj > μ*, headquarters acquires information about Zj until the posterior variance of √λjZj equals μ*.

If λj ≤ μ*, headquarters acquires no information about that mode.

Think of μ* as a residual-uncertainty floor. A mode whose prior variance lies above this floor receives attention until its posterior variance is reduced to μ*. A mode already below the floor receives no attention.

Modes that receive information are attended; the remaining modes are unattended. The attention budget ρ determines the water level and therefore n, the finite number of modes that receive attention.

The paper proves that this same rule continues to solve the Gaussian-process problem. Although the environment contains infinitely many possible modes, any finite attention budget reaches only finitely many of them.

Choose headquarters' attention capacity to see which modes it learns about and how those signals determine the posterior expectation of W0(x).

Repeat the simulation to see the role of signal noise. The same attention capacity can sometimes produce a very close estimate and sometimes a poorer one because the signal process S is itself random.

Attention
less attentionmore attention
Prior variance along the jth dimension, λj

Red bars enter the attention problem; gray bars do not.

Zj
draw E[Zj|Sj]
Headquarters' posterior expectation of Zj.
W0(x)
realized path E[W0(x)|S]
posterior mean

After simulating, move the slider to compare counterfactual attention levels. The realized state and noise draws remain fixed; only signal precision changes.

From Attention to Team Size

We have now seen how to solve the attention problem. We are therefore positioned to answer the main economic question about organization design.

Holding the adaptation-cost function fixed, the key determinant of team size is the share of total variation that headquarters leaves unresolved.

When k = 1, the team-state process is simply W0(x)/2. If μ* denotes the common posterior variance of each attended mode, the organization's objective becomes:

Increasing team size scales every mode of the team-state process—and the water level—by the same factor. It therefore does not change which modes headquarters attends to; instead, it reduces the overall coordination burden.

The set of modes headquarters attends to does not change merely because k changes.

Thus, in the coordination term for general k, the common factor 1/4 becomes 1/(4k).

Why the team-state process and its modes scale this way

A worker's optimal action is W0(x)/2. When k = 1, the team-state process is therefore

W1(x) = ½W0(x).

More generally, the continuum team-state process is

W1(u) = ½W0(ku),  u ∈ [0, 1/k].

As team size increases, the mass of team-level states falls by a factor of 1/k.

This change of scale implies that if λj is an eigenvalue of the task-state process, the corresponding eigenvalue of the team-state process is

λj/(4k).

Every eigenvalue is therefore multiplied by the same factor 1/(4k).

The water level scales by the same factor. Because both the eigenvalues and water level move together, the inequalities determining whether a mode is attended do not change with k.

In the Section 4 visualization, the eigenvalues and water level are displayed on the task-state scale, suppressing the common factor 1/4 from the k = 1 team-state process. This normalization does not affect which modes receive attention. In the organization-design problem, the full factor 1/(4k) is restored.

Putting this all together, the full objective is

After normalizing by total variance, team size depends on the fraction of variation that remains unresolved after headquarters allocates its attention. This unresolved share includes both the posterior uncertainty that remains in attended modes and all of the variation in unattended modes.

Larger teams are valuable when this unresolved share is high, because grouping workers reduces the residual variation that enters the coordination problem.

Increasing attention capacity reduces the unresolved share while leaving the marginal adaptation cost unchanged. Optimal team size therefore decreases with attention capacity until it reaches the lower bound k = 1.

Multiplying C(x,x′) by a constant leaves team size unchanged: both μ* and the λjs scale by the same constant—and hence so do adaptation and coordination losses. Total variance alone therefore does not determine organization design. What matters is how much variation attention resolves and where the remaining variation lies.

When variation increases in an unattended mode, none of that increase is dampened by headquarters through the use of its attention budget. Coordination loss therefore rises alongside adaptation loss, making the coordination benefit of larger teams more important. But the opposite happens if variation increases in a sufficiently important attended mode. Adaptation loss rises more than residual coordination loss, pushing the organization toward smaller teams.

Where complexity enters matters. More variation in unattended modes favors larger teams; more variation in sufficiently important attended modes favors smaller teams.

Jaggedness: Complexity and Organization Design

The condition on the λjs is useful mathematically, but to see what it means economically it is more enlightening to look at what the change looks like in terms of task-state sample paths.

The paper considers one particular measure of local complexity, which it calls jaggedness—informally, how "squiggly" the sample paths appear.

One way to generate greater local complexity is to slow the rate at which λj decays to zero as j grows, holding total variance fixed. In the family illustrated here, doing so reallocates variation from a few leading modes toward many higher-indexed modes, causing nearby task states to differ more sharply.

This is a form of local complexity because nearby tasks can require noticeably different actions even when their locations are very close. One economic interpretation is that expertise transports less readily across nearby tasks: even people working on similar topics may need to take meaningfully different actions.

Use the controls below to vary attention capacity and the λj decay rate. The underlying Zj draws remain fixed as the sliders move. You can therefore attribute every change in the path and optimal team size to the λj sequence and attention capacity—not to a different random realization.

Attention
less attentionmore attention
λj decay rate
slow decayfast decay

Moving the decay slider varies the covariance kernel C(x,x′) within a one-parameter family while holding total variance fixed. You are therefore choosing among environments that differ only in how their variation is distributed across modes.

Optimal team size: —
Interpreting team size as a continuous variable

The model treats k as a continuous choice. In a discrete organization with many teams, a noninteger k can be approximated by mixing adjacent integer team sizes in proportions chosen so that their average size is k—for example, k = 2.5 can be approximated by having half the teams of size 2 and half of size 3. As the worker grid becomes fine, this construction delivers the same limiting mass of teams and the same limiting team-state process. Taking f(k) to be smooth can be thought of as a reduced-form measure of average adaptation cost, rather than mechanically interpolated from costs defined only at integer team sizes. Section 3 of the paper discusses this interpretation.

What We Learned

To summarize, the following schematic illustrates the key steps in the analysis:

C(x,x′)
{(λj,φj)}
attention across modes
residual uncertainty
k*

This interactive began with a question: how does horizontal complexity shape organization design? Horizontal complexity concerns how informative the ideal action at one task location is about the ideal action at another: the less informative one location is about another, the more horizontally complex the environment.

The answer is not simply that more complexity produces larger teams. Additional variation that headquarters can substantially absorb through attention tends to favor smaller teams. Additional variation that headquarters leaves unresolved makes larger teams more valuable.

The same forces also generate predictions for other organization-design margins. One particularly useful implication is a unified explanation for why AI can push organization design in opposite directions. A technology that expands headquarters' attention tends to shrink teams. A technology that disproportionately simplifies the modes headquarters already tracks can leave the remaining complexity relatively concentrated outside attention, causing teams—and potentially hierarchies—to expand.

More on what else is in the paper

The paper considers several extensions.

First, when the organization can choose hierarchy depth, the same forces that favor larger teams also favor additional hierarchical layers.

Second, the organization can invest in simplifying its environment. The cost of changing the process is measured using the Kullback–Leibler divergence between the modified and original environments. The paper shows that more effective simplification increases team size because investment is concentrated in the leading modes headquarters attends to, leaving residual variation relatively concentrated in higher-indexed modes.

Another investment extension allows the organization to improve the adaptation problem directly—for example, through training.

Finally, the paper studies when divisions should be merged. This extension introduces cross-division interdependence and a shared attention problem, allowing the model to compare integration with separation.

What matters is not only how much complexity an organization faces, but whether that complexity falls inside or outside management's capacity to understand it.

For the complete picture, please see the full paper.