How does a computer turn a photo into numbers?

♣ Newbie

A digital photo is like a mosaic: it’s made of tiny coloured squares, the pixels, too small to see with the naked eye. When the computer opens a photo, it stores each square as three numbers: how much red, how much green and how much blue it has, each from 0 to 255. Pure red is (255, 0, 0); white, (255, 255, 255); and black, (0, 0, 0).

A photo 4,000 squares wide and 3,000 high has 12 million pixels. With three numbers each, that’s 36 million numbers in a row.

A list of numbers in a row is called a vector, and how many numbers it has is its dimension. To the computer, your photo is a vector of dimension 36 million. Everything it does with it is sums with that list.

♦ Intermediate

A vector is an ordered list of numbers, its components. With two components it can be drawn as an arrow from the centre: v = (3, 2) means “3 to the right and 2 up”. Its length comes from Pythagoras: √(3² + 2²) = √13 ≈ 3.6 (OpenStax).

A grey photo with two pixels would be an arrow like that, in a plane; with three pixels, an arrow in space. With more it can’t be drawn any more, but the sums are the same: the length is still the square root of the sum of the squares. The dimension is how many numbers the vector has.

In a colour photo each pixel carries three numbers from 0 to 255, one per channel (red, green and blue), because each takes up 8 bits (Pillow, a Python imaging library). So an open photo is a table of height × width × 3 numbers. The JPEG file takes up less space because it’s compressed, but opening it unpacks it into that table. To hand it to a neural network, it’s all put in a row: 4,000 × 3,000 × 3 = 36 million numbers.

♠ Technical

An 8-bit RGB image is a tensor of integers in {0, …, 255}^(H×W×C) with C = 3: Pillow 13.0 describes the mode «RGB (3x8-bit pixels, true color)» and says that «an 8-bit pixel has a range of 0-255». torchvision 0.29 converts it with ToTensor, from (H × W × C) in [0, 255] to a floating-point tensor of shape (C × H × W) in [0.0, 1.0]. Flattened, it’s a vector in ℝⁿ with n = H·W·C; for 4000 × 3000, n = 3.6×10⁷.

The dimension of a space is the number of vectors in any basis (Austin, Understanding Linear Algebra, def. 3.5.6). The standard basis e₁, …, eₙ has n vectors, so dim ℝⁿ = n: the number of components. The norm ‖x‖ = √(Σᵢ xᵢ²) is Pythagoras in n dimensions.

♣ Newbie

A fruit seller sells 30 apples, 12 bananas and 18 oranges in a day, at $0.50, $0.25 and $1 each. How much did they take? You multiply each quantity by its price and add: 15 + 3 + 18 = $36. That “multiply one by one and add” of two lists is the dot product, and it’s the sum a computer does most with a photo. The example comes from a maths textbook, OpenStax.

With arrows, the dot product tells you something else: how much they point the same way. If you push a trolley at an angle, only the part of the push that goes where it’s moving counts. If two arrows point the same way, the dot product comes out positive; if they cross at a right angle, zero; and if they point in opposite directions, negative.

♦ Intermediate

The dot product of a = (a₁, a₂) and b = (b₁, b₂) is a·b = a₁b₁ + a₂b₂: multiply the components one by one and add them up (OpenStax). With more components, the same. With a = (3, 1) and b = (1, 2): 3·1 + 1·2 = 5. With two photos of 36 million numbers, that’s 36 million multiplications.

With arrows it has another reading: a·b = length of a × length of b × cosine of the angle between them (OpenStax). The cosine is 1 if they point the same way, 0 if they make a right angle and −1 if they’re opposite. That’s why two perpendicular arrows give 0.

If you divide the dot product by the two lengths, only the cosine is left, which measures how alike the directions are: it’s the cosine similarity. With photos it has a nice property. If you multiply every number in a photo by 1.2, it looks lighter: the arrow gets longer but doesn’t turn, and its similarity with the original is still 1 (our own calculation, as long as no number goes over 255).

♠ Technical

In ℝⁿ, a·b = Σᵢ aᵢbᵢ. OpenStax (Calculus Volume 3, 2.3) proves with the law of cosines that a·b = ‖a‖‖b‖cos θ (theorem 2.4), and that two nonzero vectors are orthogonal if and only if a·b = 0 (theorem 2.5). In more than three dimensions it goes the other way round: the angle is defined by cos θ = a·b / (‖a‖‖b‖), and the Cauchy–Schwarz inequality, |a·b| ≤ ‖a‖‖b‖, ensures the ratio falls in [−1, 1].

PyTorch 2.14 computes cosine similarity with torch.nn.functional.cosine_similarity, which divides by max(‖x₁‖‖x₂‖, ε) to avoid dividing by zero. Since cos(x, λx) = 1 for λ > 0, cosine similarity can’t tell a photo from a lighter or darker one that only scales its values. The cost is linear: n products and n − 1 sums.

♣ Newbie

Now the fruit seller compares two shops, each with its own prices. They write the prices in a table: one row per shop and one column per fruit. That table is a matrix. To find out how much they’d pay in each shop, they multiply their shopping list by each row and add: one dot product per shop, all at once.

A matrix can also move drawings. Picture a grid painted on a rubber sheet: the matrix says where each point goes, as if you stretched or slanted the sheet, but with two rules: straight lines stay straight and the centre stays put.

Programs that rotate or stretch a photo do sums like these to decide where each pixel goes. And some matrices squash the whole drawing onto a line: after that, there’s no way to get it back.

♦ Intermediate

A matrix is a rectangular table of numbers; one with m rows and n columns is m × n (OpenStax). Multiplying a matrix by a vector means taking the dot product of each row with the vector. With A = (2 1 / 0 1) and x = (1, 2), the first row gives 2·1 + 1·2 = 4 and the second, 0·1 + 1·2 = 2, so Ax = (4, 2).

There’s a trick to reading it: the first column is where (1, 0) goes and the second, where (0, 1) goes (Austin). In the example, (1, 0) goes to (2, 0) and (0, 1) goes to (1, 1): the matrix stretches sideways and slants. With that you know where everything goes, because any point is a mix of those two.

The little square of side 1 becomes a parallelogram. With the matrix (a b / c d), its area is ad − bc: in the example, 2·1 − 1·0 = 2, double (Austin). If it’s 0, the matrix squashes the whole plane onto a line: a dimension is lost and there’s no way back. (1 −1 / 2 −2) does that. If it’s negative, the drawing comes out as in a mirror.

Pillow uses it like this to rotate or slant a photo: for each new pixel, it works out with a matrix which point of the original it comes from (plus a shift).

♠ Technical

An m × n matrix A defines T(x) = Ax from ℝⁿ to ℝᵐ, with (Ax)ᵢ = Σⱼ aᵢⱼxⱼ. Its columns are the images of the basis: A = [T(e₁) ⋯ T(eₙ)] (Austin, prop. 2.5.8). Since T is linear, T(Σⱼ cⱼeⱼ) = Σⱼ cⱼT(eⱼ), and the columns tell you everything.

In 2×2, the rotation by an angle θ is (cos θ −sin θ / sin θ cos θ), and (1 1 / 0 1) is a shear (Austin, 2.6). det A = ad − bc is the signed area of the parallelogram of the columns (3.4), and A is invertible if and only if det A ≠ 0 (prop. 3.4.12). With det A = 0 the rank drops: the image Col(A) has dimension rank A (prop. 3.5.12), and with (1 −1 / 2 −2) it’s a line.

Pillow 13.0 (ImageTransform.AffineTransform) takes «the first two rows from the inverse of an affine transform matrix»: for each output pixel (x, y), it reads the input at (ax + by + c, dx + ey + f). It uses the inverse because it walks the output photo, and the shift (c, f) makes the transformation affine, not linear.

♣ Newbie

To get dressed, first the sock and then the shoe. The other way round doesn’t come out the same. Matrices are the same: making one change and then another is equivalent to a single matrix, which you get by multiplying the two, and order matters. Rotating a photo and then stretching it sideways isn’t the same as stretching it and then rotating it.

And here’s the answer to the question at the start. A neural network that recognises cats receives the photo as a huge vector and multiplies it by matrices, one after another, with a few extra steps in between. Each row of those matrices is like a price list: its dot product with the photo gives one number. For each photo, that’s millions of multiply-and-add sums.

That’s why graphics cards suit AI so well: they’re built to do a huge number of multiplications at once.

♦ Intermediate

To multiply two matrices, each number in the result is the dot product of a row of the first with a column of the second (OpenStax). It only works if the first has as many columns as the second has rows: an m × k one times a k × n one gives an m × n one.

Applying first B and then A to a point is the same as applying the matrix AB in one go. Careful: it’s written the other way round from how it’s done (Austin). And AB usually isn’t equal to BA (OpenStax). With R, turning 90°, and S, stretching to double the width: turning and then stretching is S·R = (0 −2 / 1 0), and stretching and then turning is R·S = (0 −1 / 2 0).

A layer of a neural network multiplies its input by a matrix of weights (PyTorch). A grey photo of 100 × 100 is 10,000 numbers; a layer with 1,000 outputs uses a 1,000 × 10,000 matrix and does 10 million multiplications per photo (our own calculation). Between one layer and the next there’s a step that isn’t a multiplication; the lesson “Logistic regression or neural network?” explains what it is and what a network gains from it.

Graphics cards take lots of photos at once, one per column, and that turns everything into a single enormous matrix multiplication (NVIDIA).

♠ Technical

(AB)ᵢⱼ = Σₖ aᵢₖbₖⱼ, defined when the inner dimensions match (OpenStax, College Algebra 2e, 7.5). It’s associative and distributive, but not commutative. If T(x) = Ax and S(x) = Bx, S∘T has matrix BA (Austin, prop. 2.5.11): the one applied first goes on the right.

Each of the m·n entries is a dot product of k terms: m·n·k fused multiply-adds (FMA), that is, 2·m·n·k FLOP. NVIDIA counts it the same way in its GEMM guide, where each entry is «a dot-product of K-element vectors». For the 1,000 × 10,000 layer with a batch of 64 photos: 2 · 1,000 · 64 · 10,000 = 1.28×10⁹ FLOP (our own calculation).

PyTorch 2.14 writes the linear layer as y = xAᵀ + b, with one example per row and the weight A of shape (out_features, in_features). Because of the bias b it’s affine, not linear. NVIDIA puts the weights in A and the activations in B, one example per column: it’s the same sum transposed. Without a nonlinear σ between layers, associativity would make several layers in a row collapse into one.