Has anyone mixed linear algebra with formal language theory in this way? Let $G$ be the grammar:
$$
S \rightarrow aAb \\
A \rightarrow aA + a + \epsilon
$$

where $\epsilon$ is the empty string, $a,b$ are terminals and $S,A$ non-terminals with $S$ the start symbol.  Consider vectors of grammar rule RHS's, for instance the basic one $\nu = \begin{pmatrix} S\\A\\a\\b\\\epsilon\end{pmatrix}$.  Then $\nu$ is an eigenvector of some matrix that transforms vectors of grammar rule RHS's into the same. 

Let $T = 
\begin{pmatrix} 
0 &amp; 0 &amp; 0 &amp; aA &amp; 0\\ 
0 &amp; a &amp; \epsilon &amp; 0 &amp; \epsilon \\
0 &amp; 0 &amp; \epsilon &amp; 0 &amp; 0 \\
0 &amp; 0  &amp; 0 &amp; \epsilon &amp; 0 \\
0 &amp; 0 &amp; 0 &amp; 0 &amp; \epsilon
\end{pmatrix} $

Then if we let $0$ represent the empty set and concatenation by it equals $0$, then $\nu \rightarrow T\nu$ is an eigenvector with eigenvalue $\epsilon$.

Unrelated, but a regular language is represented by $T$ with only terminals and $\epsilon$ in the matrix.

Edit

I'm interested in any ideas.  Thanks.
The approach you're describing is the generating function approach. By solving systems of polynomial equations, one can calculate the number of words of given length (generated by any non-terminal), and through it the corresponding asymptotics.

For regular languages one can directly use linear algebra to estimate the number of words of given length. The result is described in the Wikipedia article as well as here; see also this question.