Column Vectors vs. Row Vectors

For the engineers who never seen this, |i> |j> |k> work exactly like unit vectors i,j,k, but you aren't limited to the usual 3 dimensions and 3 letters. Writing |x> in a ket like this just means "I am using x as a label for a degree of freedom in some vector space, not as the literal value x". This lets you do cool shit, like if you got n dimensions you can write a vector as sum v_i |i> ranging from 0 to n. You can also do super useful stuff like merging two vector spaces together to represent a larger system (aka a tensor product). Eg if I have a vector v = sum v_i |i> representing one subsystem and another vector w = sum w_j |j> representing some other subsystem, I can construct the vector u = sum v_i w_j |i, j> representing the whole system (sum is over both i and j).

(I know you already learned all this, I wrote it for everyone else's benefit).

Unrelated, given the webs origin at cern, HTML should have used bras as start tags and kets as end tags. <HTML| <head|head> <body| ... |body> |HTML>.

In my not-super-extensive experience, if you're programming and dealing with linear algebra problems - and you're past the MATLAB prototyping stage - then I really suggest using the BLAS/LAPACK API. You're not gunna beat decades of nerds programming ballistic missiles on punchcards. They're kinda weird and unwieldy and don't match to textbook math - but they've had a ton of thought put into them. And are made with as few footguns as possible. At least re-implementing an algo with them a few times is really educational. You'll see that it forces you to think about your memory layout and you'll realize that the back of napkin complexity calculations are actually tricky to massage into a good algorithms that use memory well. The final result with something like the Intel MKL will be way way faster than anything your abstraction can achieve

(Unfortunately I haven't found a good primer on using the BLAS well)

For the sake of completion: A vector is a type-(1,0) tensor. A linear form (or "covector" if you prefer this term) is a type-(0,1) tensor. A bilinear form is a type-(0,2) tensor.

The point of saying the type of a tensor, is to clarify what geometric object an array of numbers represents. A curse (and a blessing) of matrix notation is that it doesn't tell you this. Both a type-(0,2) and type-(1,1) tensor can be represented by matrices, but the first is used for (let's say) dot products, and the second is used for linear transformations.

The only problem with type-(m,n) notation is remembering whether "m" is the contravariant order or the covariant order. It turns out that it's contravariant, which is the opposite to what the naming would suggest. In programming, we would do {cov: m, con: n}, so we wouldn't have to remember.

This doesn't say anything regarding their practical storage, but at least conceptually vectors/covectors are a bit more than just a collection of numbers.

This distinction is pretty useful in practice when encoded in types to avoid implicitly mixing things that are conceptually different.

Horizontal vectors are forms (maps from R^n to R). Vertical vectors are vectors (elements of R^n).

Forms applied to vectors are “rows multiplied by columns”, and so: numbers.

Same thing as bras and kets, by the way.

>It's only intuitive to think of this as an array of rows given how it's laid out visually.

That's often the least important thing when designing a fast library. It's much better to optimize for common operations in your problem to be fast and thus choose the storage layout of a individual matrix instance accordingly (row-major, column-major, triangular, diagonal, block diagonal, band) so your data cache doesn't get tripped up regularily.

The actual API for creating new matrices can still be standardized and always the same. You can have the standardized API accept four column vectors for construction just as well. Just document it as such.

If you want, you can also represent the different kinds of vectors as different types, so you don't get identification problems that shouldn't exist. Also, lists of vectors are just that and there's no reason to represent both "list" and "vector" as "array". There are reasons to make the API for "vector" richer--especially since that's why the term "vector" was defined in the first place.

But for tiny 4x4 matrices, who cares about caching. But 4x4 are a minority use of matrices. Try 4096x4096.

As for OpenGL, they chose to keep the internal storage of matrices the same as it was on IRIS GL. That's a good decision for OpenGL and orthogonal to the question of how column vectors of that matrix can be accessed via an API and to the question of how row vectors of that matrix can be accessed via an API and to the question of how matrices can be created via an API.

From a mathematics standpoint, a matrix represents a linear map (a map is a function that takes a vector in space V to a vector in space W). It makes sense to be able to use matrix and linear map interchangeably as a user (i.e. use them just like you would call functions).

Firstly, almost no-one actually creates their matrices as literals. Instead they are build dynamically an usually incrementally. Nevertheless, people still think back to these very basic examples when trying to understand things, so this probably should be standard.

Secondly, these decisions matter a lot for matrix vector multiplication. The 'C' way is very efficient for computing Ax which presumes vector x to be a Column vector. When you want to compute xA instead which presumes vector x to be a row vector, you want column-major ordering for efficiency. The same works for matrix-matrix multiplication AB where you want A to be row-major and B to be column-major.

To sum up. Because of our writing direction, and the natural way to write 2-d arrays, it makes sense to have row-major ordering. That immediately means it also makes sense to use column vectors, generally speaking. There are also exceptions if you happen to know if a matrix will often be at the 'left' or 'right' side of a multiplication.

Some more interesting notes that are less relevant:

In C the matrix you defined has an incompatible type with how most matrices are stored (most people in C store rows contiguously, then define a matrix as an array of pointers to rows).

    int mat[3][3]; vs int *mat[3]

Another common ways to use matrices is element-wise, where you refer to x[i][j]. Mathematicians expect i to be the row and j to be the column. Which matches C. Whilst Fortran does it the other way around.

I’ve written godawful python extensions to feed TF or PyTorch mmapped regions to the pipeline in an efficient way. This is currently the way of things.

But half the trouble in model architecture engineering is keeping the ranks straight and having tools to debug when you messed up.

Einstein struggled with tensor math, read his letters. But he didn’t have modern computers or autodiff either. For that era, thinking in higher dimensions was a job for a genius.

I harbor a vague suspicion that some people get paid more because it’s needlessly hard.

Mathematicians use row-major indexing just like any sensible person.

Oh we can get more abstract than that.

A vector is an element of a vector space, or, if you'd like, an F-module where F is a field. A matrix is a homomorphism between vector spaces.

Mathematicians generally don't care because equations are basically pseudocode and they can make up new syntax and hand-wave edge cases away whenever they want.

the matrix is simply the layout and so R^n is column on the other side of the equals. It's arbitrary.

For computer graphics, the main advantage is that you can represent translation and projection in the matrix in addition to the other linear transformations like rotation and scaling.

For projection, the 4th vector coordinate, usually named "w", is called the homogeneous coordinate and it is typically not 1 after going through the projection matrix. As a final step, it divides all the other coordinates, so (x,y,z,w) becomes (x/w,y/w,z/w,1), x/w and y/w are the 2D screen coordinates, and z/w goes into the Z-buffer.

A matrix represents a transform of the form:

    x' = Ax + By + Cz 
    y' = Dx + Ey + Fz 
    z' = Gx + Hy + Iz 

Although this can represent scales and rotations, there's no way to represent a simple translation with this.

As proof, imagine you want to move everything 3 units along the x axis. You really just want to add 3 to x (x' = x + 3) but you can't: x' is always defined in terms of x, y and z (x' = Ax + By + Cz). There's no room for a constant.

To represent translations, then, what you really want is (x' = Ax + By + Cz + D), where D isn't multiplied by any component of the input vector, it's just D, your translation.

Well, it turns out you can do this by just adding an extra column to the matrix and using 1 for the fourth component of your vectors.

Now x' = Ax + By + Cz + Dw, where w=1, and D is your translation amount.

The full matrix then becomes

    x' = Ax + By + Cz + Dw 
    y' = Ex + Fy + Gz + Hw 
    z' = Ix + Jy + Kz + Lw 
    w' = Mx + Ny + Oz + Pw 

Now here's the trick. You can embed your three-dimensional space into a four-dimensional space at a fixed fourth coordinate (but not zero). In this new space, translation in the original three-dimensional space is a linear transform.

You may opt into some other mathematical niceties too, which are handy for applying perspective transformations, but the main takeaway is that you get efficient compositions of affine transformations.

That's not just a mechanical computational consideration. One important interpretation of matrix-matrix multiplication is composition of linear transformations. When multiplying AB, you apply multiply matrix A by each column (vector) of B, and concatenate them to form the columns of the result AB. This corresponds to applying the transformation represented by A to a set of basis vectors. This correspondence is essential to the interpretation of matrix multiplication as a linear transformation, so it's important that it work out without additional manipulation.

Matrix-vector multiplication is more or less the heart of linear algebra, so in general it needs to be an elegant sensible operation.

I understand why the "rules" of matrix multiplication make the distinction -- but what is underneath those rules? What does it mean?

I think it's because column vectors are less of a pain to write on the blackboard.

I'm being serious. There is no reason to say that columns are vectors and rows are linear functionals. The other way around is a perfectly valid convention. But I did find that when I taught linear algebra, column vectors were just easier to write on the board.

for me, the GL style (whatever you decide to call it) seems to make sense from a performance POV. I often want one axis or the translation. It's a single SIMD load since vs if they were stored the other way I'd have to load every 4th float.

Also, while you do type them out in code different from math

    x x x x
    y y y y
    z z z z
    t t t t

    xaxis = [
      .
      .
      .
      .
    ]
    yaxis = [
      .
      .
      .
      .
    ]
    zaxis = [
      .
      .
      .
      .
    ]
    translation = [
      .
      .
      .
      .
    ]

    matrix = [xaxis, yaxis, zaxis, translation]  // 4 columns side by side