text:
Encode - Shuffle - Decode
Challenge
Your task is to encode an integer as a string of ASCII characters, then successfully decode it after said string has been randomly shuffled.
You will write two programs/functions, which will be referred to as Encoder and Decoder.
Encoder

Input: an integer \$n\$ in the range \$[0,2^{31}-1]\$.
Output: a string \$s\$ of ASCII characters (not necessarily printable).

Decoder

Input: a random permutation \$s'\$ of the string \$s\$.
Output: the integer \$n\$.

Scoring
Let \$A\$ be the maximum length of \$s\$ across all possible values of \$n\$. If the Encoder acts non-deterministically (which is allowed, see below), then the \$A\$ will be the maximum length of \$s\$ that may occur (possibly \$\infty\$).
Let \$L_E\$ be the length of the Encoder in bytes and \$L_D\$ the length of the Decoder in bytes.
Then your score is \$A\cdot(L_E+L_D)\$.
Victory is awarded to the submission the lowest score.
Time limit
There is a somewhat arbitrary time limit of 1 minute on the execution time of both the Encoder and the Decoder for a single testcase (i.e. a single value of \$n\$).
The goal is to avoid solution that find that brute-force the encoding by enumerating all sequences with certain properties. If your solution does something more clever than that, it will most likely fit the time constraint and will be considered valid. Likewise, if it works on TIO for some randomly selected values of \$n\$ it will be considered valid. Otherwise I will test it on my machine, but note that if your solution is pure brute-force it will almost certainly fail.
Rules

The Encoder and the Decoder must be written in the same language.
The Decoder must output the correct integer \$n\$ for every possible permutation \$s'\$ of the string \$s\$ returned by the Encoder.
The Encoder and the Decoder are not allowed to share information in any way (e.g. by means of global variables or files).
The output of the Encoder need not be deterministic (that is, the same input \$n\$ may produce different output strings if the Encoder is run multiple times), but the Decoder must always guess the correct integer \$n\$.
The Encoder and the Decoder may take and return the integer \$n\$ in any convenient way (e.g. if \$n=14\$ it is fine for the input to be 14, "14" or [1,4]).
The Encoder may output the string \$s\$ either by printing it on stdout or by returning a string, a list/array of characters or a list/array of integers in the range \$[0,127]\$; note that the Decoder will receive as input a permutation of \$s\$ as returned by the Encoder, so it should accept the string \$s'\$ in the same format as \$s\$.
Standard loopholes are forbidden.
If possible, explain how your code works and why the score you claim is correct.

Example

Assume \$n=14\$.

The Encoder receives 14 as input. It may output "qwerty".
The Decoder receives a permutation of "qwerty" as input, for instance "tweyqr". It must output 14 (in any convenient format).

The Encoder could have returned [113,119,101,114,116,121] as well, in which case the Decoder would have received (for instance) [116,119,101,121,113,114].
Note that the string returned by the Encoder may also include non-printable ASCII characters (but always in the range [0x00, ..., 0x7F]).

Surely the output length can't be infinity, you can't shuffle an infinite string

@H.PWiz No it can't, but the lenght may be unbounded if the Encoder is non-deterministic

"The Encoder and the Decoder are not allowed to share information in any way" Does this include helper functions? i.e. A custom function that calculates N factorial plus three (random example)

Can our Encoder return an empty string/list?

@pizzapants184 The empty string/list is certainly a valid return value. As for shared helper functions, I would say they are not allowed, but I am unsure about standard rules, so please let me know if there is a consensus approving them.

What if the decoder and encoder are the same? Do you have to count the bytes twice or just once?

@Kroppeb Yes, as of now the rules say you should count the bytes twice. I'm very interested in seeing a submission with two identical programs, though.

Can we print leading zeroes for the decoder?

@Jo King Yes you can.

Jelly, (17 bytes + 18 bytes) × length 6 = 210 points
b36µỤỤ+×3µŒ¿b3U+Ṣ
Ṣ:3_J
Ṣ%3Uḅ3œ?Çḅ36
Try it online! (or with extra debug info)
Having had a go at solving this challenge aiming at the stated victory condition, I thought it'd be interesting to go for a hypothetical alternative victory condition: code-golf given a minimum possible maximum length for the output.
Explanation
Encoding
The first step in the encoding is to represent the input as base 36 (b36). 366 = 2176782336 > 2147483647, so there will be at most 6 digits in the result, each of which is in the range 0–35.
Next, we transform this into a representation that contains 6 different digits. There are multiple possible algorithms for this, but the one used here is to add 1 to the smallest digit, 2 to the second-smallest, 3 to the third-smallest, and so on. This means that if two digits were the same, one of them will be arbitrarily considered smaller, and thus they'll become different; and obviously, this algorithm can't cause two different digits to become the same. In order to represent this in Jelly, we use Ụ ("sort indexes by values") to get a list of the indexes in sorted order; Ụ again to invert that, effectively mapping each element of the original to its position in sorted order; and µ…+ to add the original to the new list. The result is a representation of the input number as six different digits in the 1–41 range (minimum 0+1, maximum 35+6).
We then split this into yet another form: a sorted list of digits in the 1–41 range, alongside a number from 1 to 720 that represents which of the 720 possible permutations the digits were in. (The Œ¿ and Ṣ extract the permutation number and sorted list respectively.)
Finally, we convert the number from 1 to 720 into base 3 (b3), reverse it (U), and encode the six base 3 digits and six 1–41 digits via packing them each into a single ASCII character using reverse divmod (the value of the character mod 3 is the base 3 digit, the value divided by 3 is the 1–41 digit). The possible range of results is (1×3)+0=3 at minimum, and (41×3)+2=125 at maximum, fitting within our ASCII range. The packing is done via ×3 and +, together with an additional µ to make sure that every command operates on the right bit of data. (There's a bit of a golfing trick here, in that we do the multiplication by 3 before extracting the permutation; that saves the need to spend a byte on a grouping character.)
Incidentally, the reason to reverse the base 3 number is because it may have fewer digits than the 1–41 number. (It can't have more; the smallest number for which n! > 3n is slightly above 6.) Jelly effectively inserts trailing zeros when adding together two numbers of different lengths, in order to make them match up; trailing zeroes would affect the interpretation of the number, but leading zeroes wouldn't, so the reverse is used to ensure that the extra zeroes end up somewhere that won't mess up our answer.
Decoding
The first step in decoding is to extract the two numbers (the base 3 number and the 1–41 number). We can get their digits easily enough with division (:3) and modulo (%3) respectively, but how to know which order they were in? Well, the 1–41 number had its digits in sorted order, and digits in corresponding positions of the two numbers were stored in the same characters; thus, we can work out which order the 1–41 number's digits were shuffled into (by looking at their relative values) and know that the base-3 number's digits must have been shuffled the same way. In fact, because characters of our ASCII encoding sort the same way as the 1–41 number's digits (these were all distinct and they're more significant than the base 3 number's), we can simply put the characters of the encoding back into order with a sort Ṣ. So both extractions start with Ṣ, followed by %3 or :3 as appropriate.
While the 1–41 number's digits are still in sorted order, we have a very convenient/terse way to go back to the 0–35 digits of base 36; just subtract 1 from the first, 2 from the second, 3 from the third, and so on. In Jelly, we can do that with _J ("subtract index").
Meanwhile, in the other branch of the decode, we reverse the base 3 number's digits back into order (U), and convert it from base 3 back into a permutation index with ḅ3.
We can then combine the two branches with œ?Ç; œ? means "permute given this permutation index", and Ç means "the result of applying the line above", i.e. it's what tells Jelly to run both lines separately on the same input.
What we have now is the digits of the original number, in base 36 (due to the _J), and in the original order (due to the œ?), so we can simply do a ḅ36 to convert back from base 36 into a single integer.
Commentary
The TIO! link above uses 312699167 as the number to encode. This number in base 36 is [5, 6, 6, 8, 7, 35], and thus shows off all aspects of the encoding: the 35 tests the limit of the 0–127 range we have; the duplicate 6s test the resolution of identical digits in the original base 36; and the fact that the digits are almost (but not quite) sorted means that the permutation number is very small, giving it many fewer digits than the base 36 number, and thus showing the need to reverse it before adding it to the original.
It's really convenient how all the constants here fit together. 366 is only just high enough to fit 231, 36 is only just high enough to fit 6!, and (36+6)×3 is only just high enough to fit within the 128 possibilities we have. (The last constraint here is the least tight, because we could use 0-indexing rather than 1-indexing to make use of characters in the 0-2 range. Still, that'd only give enough room to use 37 as the base rather than 36.)

Jelly, (4 3 bytes + 6 5 bytes) × length 8 = 80 64 points
b⁴ÄṢŻIḅ⁴
Try it online!
Jelly, (2 1 byte + 4 3 bytes) × length 10 = 60 40 points
ÄṢŻI
Try it online!
Explanation
Solution 1
This is using a different algorithm from most of the other answers. We start by encoding the value in hexadecimal (b⁴), as with the other answers, then take a cumulative sum (Ä). Each input will clearly give a different output (as both these operations are reversible), and given that the hexadecimal encoding will contain at most 8 digits whose maximums are 7 (for the 8th-last digit) and 15 (for the last to 7th-last digits), the maximum number in the output list will be 7+(7×15) = 112, less than the 127 required by the question. Additionally, the output will necessarily be in sorted order, allowing us to reverse the shuffle.
For the decoder, we first reverse the shuffle with a sort (Ṣ); then reverse the cumulative sum, by prepending a zero (Ż) and taking the difference of consecutive pairs (I); then convert back from hexadecimal (ḅ⁴).
Solution 2
The question actually allows us to take the input as a list of (presumably decimal) digits, so we can "cheat" by simply removing the base conversion; the maximum number used in the output will then be 2 + (9×9) = 83 (actually 82 because 2999999999 is out of range, so the worst possible input is 1999999999). The resulting encoding is pretty terrible as encodings for this problem go, but it has the advantage of being very terse to generate, which outweighs the verbosity of the encoding.
This answer feels so much like cheating that it's not my primary solution for this problem, but it seems like it's worth adding because it technically complies with the rules and produces a better score.
Commentary
I have some algorithms in mind for getting below length 8, but it seems unlikely you could implement a length-7 algorithm in ≤9 bytes (non-cheating) or ≤5 bytes (cheating), so by the scoring in the question, this is likely the best way to do it. (I might have a go at a solution for the alternative "minimize the length of the encoding" challenge anyway, though, just for fun.)
Unlike some of the solutions, the use of 16 as the base here isn't critical; there are plenty of other numbers that would work for a length 8 solution (e.g. 18). I picked 16 for the first solution simply because Jelly has a 1-byte way to represent that, and other viable bases would need to use up multiple bytes from the program. Of course, the second solution needs to use 10 as the base in order to exploit the loophole.
Thanks to @Dennis for pointing out some newer Jelly commands that made this algorithm even terser to write.

Ä is short for +\, Ż is short for 0;.

Shakespeare Programming Language, 10 * (264 + 494) = 8650 7910 7580
Encoder: 264 bytes
,.Ajax,.Ford,.Act I:.Scene I:.[Exeunt][Enter Ajax and Ford]Ajax:Open mind.Be you nicer zero?You be the sum ofyou the product ofthe sum ofI a big big pig the sum ofa big big big big cat a big big pig.If soSpeak thy.Ford:You are the sum ofyou a cat.If soLet usAct I.

Try it online!
Decoder: 494
,.Ajax,.Ford,.Page,.Act I:.Scene I:.[Exeunt][Enter Ajax and Ford]Ajax:Open mind.Is you nicer a pig?If soRemember you.If soLet usAct I.Scene V:.Ford:Remember I.Ajax:Recall.Is you worse the sum ofPage twice the sum ofa big big cat a cat?If soYou be the difference betweenyou Page.If soOpen heart.If notlet usScene V.Scene X:.Ford:Recall.Ajax:Be I nicer zero?If soRemember I.If soLet usScene X.[Exit Ajax][Enter Page]Ford:You be the sum ofyou twice twice the sum ofa big big cat a pig.Let usAct I.

Try it online!
This was a thing.
The encoder encodes each digit as the digit plus the index of the digit times twelve. The decoder stores all the input in the Ford's memory and then loops over a counter, outputting then deleting each digit lower than the counter*12 + 10.
Explanation:
Encoder
,.Ajax,.Ford,.Act I:.Scene I:.      Boilerplate introducing the two characters
[Exeunt][Enter Ajax and Ford]       Enter the two characters Ajax and Ford
                                    Ford will be handling the input
                                    Ajax will be the counter
Ajax:Open mind.                     Set Ford to the next character of input
Be you nicer zero?                  Check if it is EOF
You be the sum of                   Set Ford to the sum of
    you                             His original value (48 to 58)
    the product of
          the sum of
              I                     Ajax's value
              a big big pig         Minus 4 (this handles the offset of 48)
          the sum of                Multiplied by
              a big big big big cat 2^4
              a big big pig.        + -2^2 = 12
                                    This essentially sets Ford to (F+12*(A-4))
If soSpeak thy.                      If not EOF, print Ford's value
Ford:You are the sum ofyou a cat.   Increment Ajax's value
If soLet usAct I.                   If not EOF, Repeat again.

Decoder
,.Ajax,.Ford,.Page,.Act I:.Scene I:.  Boilerplate introducing three characters
                                      Ajax is the spare stack
                                      Ford is the storage stack
                                      Puck is the counter, increasing by 12
[Exeunt][Enter Ajax and Ford]            Enter Ajax and Ford onto the stage
Ajax:Open mind.                          Get the next character of input
Is you nicer a pig?                      If not EOF
If soRemember you.                         Store the value in Ford's memory
If soLet usAct I.                          And repeat the loop
Scene V:.                                Otherwise move to the next scene
Ford:Remember I.                         Store Ford's value (initially -1 from EOF) in Ajax's memory
Ajax:Recall.                             Get the next value from Ford's memory
Is you worse the sum of                  Is the value smaller than
        Puck                                  Puck's value
        twice the sum ofa big big cat a cat?  + 10 ?
                                              i.e. is it the next digit?
If soYou be the difference betweenyou Puck.   If so, subtract Puck's value from Ford
If soOpen heart.                              And print Ford's value
If notlet usScene V.                     If that was not the digit, repeat
Scene X:.
Ford:Recall.                             Get the next value from Ajax's memory
Ajax:Be I nicer zero?                    Until the -1
If soRemember I.                         Returning the values to Ford's memory
If soLet us Scene X.                     Repeat until Ajax's memory is exhausted
[Exit Ajax][Enter Page]                  Swap Ajax and Page
Ford:You be the sum of                   Set Puck's value to
              you                        Puck +
              twice twice the sum of     2*2*(
                           a big big cat      4
                           a pig.             -1) = 12
Let usAct I.                             And start from the beginning again, having removed one number

Python 2.7, 31 * (52 + 37) = 2759
Encoder (69 52 bytes):
lambda n:[chr(i)if n&(1<<i)else""for i in range(32)]

Decoder (41 37 bytes):
lambda s:sum([1<<(ord(c))for c in s])

Stores the all the non-zero bits in the input number as ascii values. The value of the ascii character stores the position of the set bit. For example the value 'a' would mean that the 97th bit is set.
A few improvements, thanks to @Delfad0r
Try it online!

Welcome to PPGC! You can drop the e =  and the d =  at the beginning - anonymous functions are perfectly fine. Also, note that the problem statement clearly says that the Encoder may return a list of integers instead of characters, so you can avoid the conversion integer->character->integer. Moreover, you can use n&(1<<i) instead of n&(1<<i)>0 and save 2 bytes. Finally, the upper bound for i (127) is way too much, 32 is enough and saves 1 byte.

Please state your score according to the Scoring section in the problem statement.

@Delfad0r Is the scoring correct now? And thanks for the tips.

I think the score is (52+37)*31=2759 since the longest is when all 31 bits are set.

Encoder can be lambda n:[chr(i)*(n&1<<i>0)for i in range(32)] to save 6 bytes.

Python 3, 8 * (45 + 38) = 664
Encoder (45 bytes):
lambda n:[16*i+(n>>4*i)%16 for i in range(8)]

Decoder (38 bytes):
lambda l:sum(x%16<<x//16*4 for x in l)

Try it online!

You can remove the spaces before "for", lambda l:sum(x%16<<x//16*4for x in l) works just fine :)

This does not work. Output is not plain ASCII (in the range 0..127)

@GB my mistake. I broke it with my last edit. Reverting now

save 3 bytes in the encoder: lambda n:[n>>4*i&15|i<<4for i in range(8)] and 1 in the decoder: lambda l:sum(x%16<<x//16*4for x in l) for a total score of 632

Stax, score 8 × (10+9) = 152
Encoder, 10 bytes
Ç·MÉJ'♀τ│½

Run and debug it
16|E{i16*+m Full program, implicit input
16|E        Get hexadecimal digits
    {     m Map:
     i16*+    Add 16 * loop index
            Implicit output as string

The encoder outputs the string in an increasing order.
Decoder, 9 bytes
üL∟n╫k∞‼9

Run and debug it
o{16%m16|E Full program, implicit input
o          Sort string
 {16%m     Module each element by 16
      16|E Interpret as array of hex digits

05AB1E, 8 maximum length * (8 + 7) bytes = 120
Encoder (8)
hεHN16*+

Try it online!
Decoder (7)
{16%hJH

Try it online!
Uses the same technique as wastl and Jonathan.

JavaScript (ES6), 8 * (40 + 32) = 576
The encoder outputs an array of \$0\$ to \$8\$ integers. The decoder takes the same format as input.
Encoder (40 bytes)
E=(n,k=0)=>n?[k|n&15,...E(n>>4,k+16)]:[]

Decoder (32 bytes)
s=>s.map(c=>s|=c%16<<(c/4&~3))|s

Demo
Try it online!
How?
The input is divided into 8 blocks of 4 bits and each block is encoded with 1 among 16 possible characters. The most significant bit of the last block is never set.
       3222222222211111111110000000000
bit:   0987654321098765432109876543210
       \_/\__/\__/\__/\__/\__/\__/\__/
block:  7  6   5   4   3   2   1   0

block #0 is encoded with char. 00 to 0F (NUL to SI)
block #1 is encoded with char. 10 to 1F (DLE to ES)
block #2 is encoded with char. 20 to 2F (' ' to '/')
block #3 is encoded with char. 30 to 3F ('0' to '?')
block #4 is encoded with char. 40 to 4F ('@' to 'O')
block #5 is encoded with char. 50 to 5F ('P' to '_')
block #6 is encoded with char. 60 to 6F ('`' to 'o')
block #7 is encoded with char. 70 to 77 ('p' to 'w')

Jelly, (8 + 9) bytes * 8 max-length = 136
b⁴+J’Ɗ⁴¡

Encoder (footer formats the list as Python would for clarity)
Ṣ_J‘Ɗ⁴¡ḅ⁴

Decoder
It's theoretically possible to have a max-length of six, can that be done in 22 bytes or fewer?
It's impossible with a max-length of five since \$\sum_{i=0}^{i=5}\binom{127+i}{127}=321402081< 2^{31}-1\$
How?
Since \$2^{31}-1\$ is encodable as 8 hexadecimal digits (7fffffff or [7,15,15,15,15,15,15,15]) we may then add the zero-based index of each hex-digit multiplied by 16 to ensure such a conversion is always in sorted order while keeping even the rightmost value in-bounds (i.e. [7,15,15,15,15,15,15,15] + [0,16,32,48,64,80,96,112] = [7,31,47,63,79,95,111,127]). Decoding is then reversing this same process.
Encoder:
b⁴+J’Ɗ⁴¡ - Link: integer, n    e.g. 1234
 ⁴       - literal 16               16
b        - convert to base          [4,13,2]
       ¡ - repeat...
      ⁴  - ...16 times:
     Ɗ   -   last 3 links as a monad:
   J     -     range of length        [1,2,3]     iter 2    iter 3    ...  iter 16
  +      -     add                    [5,15,5]   [5,16,7]   [5,17,9]  ...  [5,30,35]
    ’    -     decrement              [4,14,4]   [4,15,6]   [4,16,8]  ...  [4,29,34]
         -                                                                 [4,29,34]

Decoder:
Ṣ_J‘Ɗ⁴¡ḅ⁴ - Link: list of integers   e.g. [29,34,4]
Ṣ         - sort                          [4,29,34]
      ¡   - repeat...
     ⁴    - ...16 times:
    Ɗ     -   last 3 links as a monad:
  J       -     range of length           [1,2,3]
 _        -     subtract                  [3,27,31]   [3,26,29]   [3,25,27]  ...  [3,12,1]
   ‘      -     increment                 [4,28,32]   [4,27,30]   [4,26,28]  ...  [4,13,2]
        ⁴ - literal 16                    16
       ḅ  - from base                     1234

"hexadecimal digits", sure. ("digits using plain hexadecimal" is longer, and "digits" alone implies decimal.)

I changed it even though it should have been obvious from the context as I then immediately referred to hex-digits.

Your calculation is off by one: there are 321402081 combinations with replacement with a max length of 5, and 7177979809 with a max length of 6.

@AndersKaseorg oops, so it is - so it is possible with a max-length of 6... giving 22 bytes to play with!

Shakespeare Programming Language, 31 * (472 + 383 379 344) = 26505 26381 25296
Previous score: 16909322 * (246 + 217) = 7829016086
This is still very high, but it's the lowest I can think of right about now.
Encoder:
,.Ajax,.Ford,.Act I:.Scene I:.[Enter Ajax and Ford]Ajax:Remember a pig.Ford:Listen tothy.Scene V:.Ajax:Remember the remainder of the quotient betweenI a big cat.Ford:You be the quotient betweenyou a big cat.Be you nicer zero?If solet usScene V.Remember a pig.Scene X:.Ajax:Recall.Ford:Am I worse zero?If notremember I.If notlet usScene X.Ajax:You zero.Scene L:.Ford:Recall.Ajax:You be the sum ofyou a cat.Am I nicer zero?If sospeak thy.Am I worse zero?If notlet usScene L.

Try it online!
Decoder:
,.Ajax,.Ford,.Page,.Act I:.Scene I:.[Exeunt][Enter Ajax and Ford]Ajax:You cat.Ford:Open mind.Remember the sum ofyou I.Scene V:.Ajax:Am I nicer a cat?If soyou be twice you.Ford:If soyou be the sum ofyou a pig.If solet usScene V.[Exit Ford][Enter Page]Page:Recall.Ajax:Am I worse a cat?If notyou be the sum ofyou Ford.If notlet usAct I.Open heart

Try it online!
Basically, if the string contains a character with the ASCII code (n+1), the nth binary digit is set.

344 bytes for the decoder

Charcoal, score 10 * (10 + 15) = 250.
Uses decimal; previous base 16-based solution scored 328 296 264.
May output nonprintable characters. In particular, character 10 is tricky to input to Charcoal.
Encoder, 10 bytes:
⭆⮌Ｓ℅⁺Ｉι×χκ

Try it online! Link is to verbose version of code.
Decoder, 15 bytes:
ＩΣＥＳ×﹪℅ιχＸχ÷℅ιχ

Try it online! Link is to verbose version of code.
Version using a list of integers scores 360 296 (base 16; decimal would score 310):
Encoder, 19 bytes:
ＮθＩＥ⁸⁺﹪÷θＸ¹⁶ι¹⁶×¹⁶ι

Try it online! Link is to verbose version of code.
Decoder, 18 bytes:
ＩΣＥＥ⁸Ｎ×﹪ι¹⁶Ｘ¹⁶÷ι¹⁶

Try it online! Link is to verbose version of code.
Version using printable characters scores 360 (was 416 384 368 in base 16):
Encoder, 19 bytes:
⭆⮌Ｓ℅⁺Ｉι×χ⁺κ×⁵⊕×³÷κ⁵

Try it online! Link is to verbose version of code.
Decoder, 17 bytes:
Ｆθ⊞υ⌈Φθ¬№υκ⭆υ﹪℅ιχ

Try it online! Link is to verbose version of code.

Python 3, (208 bytes + 200 bytes) * 6 length = 2448
Try it online! (contains both, the extra byte is the newline between them).
-4 bytes (-24 score) by utilizing the empty list (which allowed more things to start at 0)
Encoder (208 bytes)
def E(n,h=128):
    T=lambda n,d:n*T(n+1,d-1)//d if d>1else d and n or 1
    d=l=0
    s=[]
    while n>=T(h,d):
        n-=T(h,d)
        d+=1
    for i in range(d):
        while n>=T(h-l,d+~i):
            n-=T(h-l,d+~i)
            l+=1
        s+=[l]
    return s

Decoder (200 bytes)
def D(s):
    T=lambda n,d:n*T(n+1,d-1)//d if d>1else d and n or 1
    s.sort()
    l=0
    d=len(s)
    n=sum(T(128,D)for D in range(d))
    for i in s:
        for j in range(l,i):
            n+=T(128-j,d-1)
        l=i
        d-=1
    return n

Observations:

Shuffling can be losslessly reversed for strictly non-increasing (i.e. sorted) lists.

Strictly non-increasing numerical lists of the same length can be totally ordered (as they are in Python).

We can define that lists are ordered by length first to form a total order of all sorted lists.

We can form an indexable sequence of these lists if we define that the only valid values in a list are integers from 0 to 127 inclusive (i.e. there exist a finite number of valid lists with length L).

Strategy:

Encoder: Given a number N, find the Nth valid strictly non-increasing list.

Decoder: Given a (shuffled) valid list, sort it and return its index in the sequence of valid lists.

Common code explanation:

T=lambda n,d:n*T(n+1,d-1)//d if d>1else d and n or 1

Compute the nth d-simplex number

For d=0, always 1

For d=1, n (the number of dots in a line of dots with length n)

For d=2, \$\sum_{i=1}^{n} i\$, (the number of dots in a triangle of dots with side length n)

For d=3, \$\sum_{j=1}^n \sum_{i=1}^{j} i\$,  (the number of dots in a tetrahedron of dots with side length n)

Encoder explanation:

def E(n,h=128): d=l=0, s=[]

n is the input number, h is the "high value" (i.e. the highest number allowed + 1), d is the length the output will be, s is the output, l is the "low value" (starting at 0, explained more later)

while n>=T(h,d):, n-=T(h,d), d+=1

There are T(h,d) valid length-d lists, and our calculation is easier if n is an index relative to the list [0]*d (at index 0) instead of an actual index, so decrement n accordingly. This also adjusts d (the length)  to be correct for the given n.

for i in range(d):

Effectively: "for the i+1th number in the list"

This is where I will explain l, the "low value"

After a number has been put into the list, no number less than it can be put in the list (to keep it sorted), so l is the last number that was added to the list.

while n>=T(h-l,d+~i):, n-=T(h-l,d+~i), i+=1

If n is too big to be encoded with an l at this "digit", then adjust n accordingly and increment l

s+=[l]

Encode n with an l at this "digit".

At first, we have h options for what "digit" to put in next, but once we put in a "digit" (which is assigned to l), we are limited to h-l options for the next "digit".

At first there were T(h,d) valid lists, but we have added a "digit" l, decreasing the number of "digits" left to d-1 and the number of valid next "digits" to h-l, so the number of valid lists after this is T(h-l,d-1)

Decoder explanation:

def D(s):, s.sort(), l=0, d=len(s)

s is the (shuffled) input list, so s.sort() it; l is the "low value" (h the "high value" is just literal 128s in the code to save bytes), n is the output number, d is the length.

n=sum(T(128,D)for D in range(d))

Adjust n to the point in the sequence of [0]*length

for i in s:

For each digit:

for j in range(l,i):, n+=T(128-j,d-1)

Adjust n to the point in the sequence of [...prevdigits, thisdigit, 0...]

l=i: Set the "low value" to the most recent digit

d-=1: Decrement the length since we used a digit

return n: After n has been adjusted for all of the digits, it is the right number; return it.

Sorry if this isn't clear, but here's my original nongolfed debug version Try it online!, which doesn't use the empty list, so is 1 off from all numbers used in this version

Ruby, (36+29 bytes)*8, score 520
Encode:
->n{(0..7).map{|x|(n>>x*=4)%16+x*4}}

Try it online!
Decode:
->a{a.sum{|x|x%16<<(x/4&28)}}

Try it online!
How it works:
The number is encoded using 4-bit chunks and a 3-bit index.
Decoder takes the input array and puts every nibble into its place again.

Gol><>, 8 * (14 + 13) = 216
Encoder Try it online!, 14 bytes:
I8FfPSD8*L+o|;

Decoder Try it online!, 13 bytes:
iEh8SD4*2$X*+

Since this can output unprintable ascii characters, thus messing with the decoder, there is now a version using numbers in the output / input:
Encoder Try it online!, 14 bytes:
I8FfPSD8*L+N|;

Decoder Try it online!, 13 bytes:
IEh8SD4*2$X*+

Encoding:
The encoding works by breaking the given number into 8 x 4bit chunks. These chunks are then shifted right by 3 bit and the original location of the chunk is appended on the end as a number between 0 and 7. Thus the encoding looks like this:
0AAAABBB
 |__|    -> the original part of the number
     |_| -> the position of the chunk inside the original number 0 = LSB and 7 = MSB

Brachylog, 17+18 bytes * 8 length = 280
Encoder:
ḃ₁₆I∧7⟧₁;Iz₁~ḃ₁₆ᵐ

Decoder:
ḃ₁₆I∧7⟧₁;Iz₁~ḃ₁₆ᵐp

A p can be added to the end of the encoder with no effect.
The decoder is run by putting the (shuffled) result as the output and getting the original number in the input.
If there would be a (properly implemented) cumulative sum predicate the score could drop down to 20
Try it online!

@Delfad0r adding the p to the encoder would make it be the same code for encoding and decoding

05AB1E, score: (2 + 2 bytes) * 11 maximum length = 44
Encoder (2 bytes):
.¥

Try it online.
Decoder (2 bytes):
{¥

Try it online.
Input of the encoder and output of the decoder are a list of digits.
Port of @ais523's 2nd Jelly answer.
Explanation:
.¥    # Undelta (automatically prepends a 0)
      #  i.e. [3,0,4,7,8,2,0,1,9] → [0,3,3,7,14,22,24,24,25,34]

{     # Sort
      #  i.e. [14,7,22,25,24,3,0,24,34,3] → [0,3,3,7,14,22,24,24,25,34]
 ¥    # Deltas
      #  i.e. [0,3,3,7,14,22,24,24,25,34] → [3,0,4,7,8,2,0,1,9]

Because .¥ prepends a zero to the output, the length of the output is the length of the input + 1. Since \$2^{31}-1\$ has a length of 10 digits, the maximum length of the output is 11.

Perl 6, 10 * (10 + 12) = 340 220
Encoder:
{^@_ Z~@_}

Decoder:
{.sort X%10}

Try it online!
The encoder function zips each digit with the 0-index of the number. Then the encoder sorts the list of numbers and gets the modulo by 10, in other words the second digit of the number.
The total is 10, since that's the maximum length of 231-1.

Haskell, 10*(23+51) = 740
Here's a program that encodes, shuffles, decodes and validates values: Try it online!
Encoder, 23 bytes
zipWith((+).(10*))[0..]

Try it online!
Decoder, 51 bytes
map snd.sortOn fst.map(`divMod`10)
import Data.List

Try it online!
Explanation
Since we're allowed to use input as decimal digits, we'll use that.. The encoder maps each digit that occurs to 10*index + digit, note that all digits will be in [0..9] so we can reverse the above by using divMod. After restoring the indices and the digits it's only a matter of sorting by the indices and getting rid of them.
The solution is expected to work for values up to \$2^{31} - 1 = 2147483647\$ which is 10 digits long, so the maximum code-point we get will be \$9 \cdot 9 = 81 < 128\$. Also each digit will be converted to a "character", so we'll end up with a maximal length of 10.

Husk, 10*(7+8) = 150
Straight port of my Haskell solution only with the observation that \$10 \cdot 9 = 90 < 128\$ (Husk's N is 1-based):
Encoder, 7 bytes
zo+*10N

Try it online!
Decoder, 8 bytes
m→Ö←m‰10

Try it online!

APL (Dyalog Unicode), \$L_E+L_D=36; A=8 \rightarrow 288\$.
d←{16⊥n-16×⍳≢n←⍵[⍋⍵]}
e←(⊢+16×⍳∘≢)16⊥⍣¯1⊢

Try it online! (contains 5 extra bytes for the assignments and the newline).
Uses ⎕IO←0
How:
(⊢+16×⍳∘≢)16⊥⍣¯1⊢ ⍝ Encoder; input 1234
          16⊥⍣¯1⊢ ⍝ Convert input to base 16 → 4 13 2
      ⍳∘≢          ⍝ [0..length of the list-1] → 0 1 2
   16×             ⍝ times 16 → 0 16 32
 ⊢+                ⍝ plus the original list → 4 29 34

{16⊥n-16×⍳≢n←⍵[⍋⍵]} ⍝ Decoder; input ⍵ → 34 4 29
              [⍋⍵]  ⍝ Grade ⍵ up → 2 0 1
             ⍵      ⍝ Index ⍵ with that list → 4 29 34
           n←       ⍝ assign that to n
      16×⍳≢         ⍝ 16×[0..length(n)-1] → 0 16 32
    n-              ⍝ subtract that from n → 4 13 2
 16⊥                ⍝ Decode from base 16 → 1234

PHP, 8*(44+53) = 776
encoder, 44 bytes:
for(;$n=&$argn;$n>>=4)echo$n&15|16*$i++," ";

prints space separated list of integers. Run as pipe with -nR.
maximum 8 bytes with 4 data bits (lower nibble) and 3 weight bits (upper nibble).
Simply put:
Put each hex digit in an own character and use the upper half of the byte to store the digit´s position.
example:
1457893891 (0x56e5b203) will turn into
0x03, 0x10, 0x22, 0x3b, 0x45, 0x5e, 0x66, 0x75
→ 3 16 34 59 69 94 102 117
decoder, 53 bytes:
while($i++<8)$n+=(15&$x=$argv[$i])<<($x>>4)*4;echo$n;

or
while($i++<8)$n+=(15&$x=$argv[$i])<<($x/4&~3);echo$n;

or
for(;$i<9;$x=$argv[++$i])$n+=$x%16<<($x/4&~3);echo$n;

take integers from command line arguments. Run with -nr.

Try them online.

Python 2, 10*(68+54) = 1220
e=lambda n:"".join(chr(int(`i`+j))for i,j in enumerate(`n`)if j<'L')
d=lambda s:int("".join(`ord(c)%10`for c in sorted(s)))

Try it online!
EDIT: Thanks to Jo King for the pointers - not sure why I was offsetting by 32, in retrospect.
Encodes the position and value of each place as a single character, starting with [space] (position 0, value 0) the NUL byte 0x0.
Decodes by:

sorting the string (Python will sort characters by their ordinal value)
converts each character into its ordinal value
takes the last digit of each ordinal integer
joins the integers into a string
converts the joined string back into an int

Do you need the 32 offset? Also, [-1] could be %10 instead, in the right place

C (gcc), 10*112 = 1120
c,i;e(i,s)char*s;{for(c=1;i;c+=10,i/=10)*s++=c+i%10;*s=0;}
d(char*s){for(i=0;c=*s++;i+=--c%10*pow(10,c/10));s=i;}

Try it online!
I have global variables, but they are not actually passing any info between two functions. Variable declaration for c is used in both functions saving me 2 bytes in code length.
A version that uses printable ASCII only for a 3 5 byte penalty is here:
c,i;e(i,s)char*s;{for(c=32;i;c+=10,i/=10)*s++=c+i%10;*s=0;}
d(char*s){for(i=0;c=*s++;i+=(c-=32)%10*pow(10,c/10));s=i;}

Thanks @ceilingcat for 70 points improvements.


text:
Mid-Autumn Festival gambling game
Tomorrow is the Mid-Autumn festival, and in the spirit of that holiday, I will introduce a gambling game that we (people from Xiamen) play during the holiday!
Rules
The game is played with six 6-sided dice. Different combinations of numbers have different ranks, with a special emphasis on fours and ones. Your job is to write a program/function that will rank the hand, given a roll of 6 dice. Here are the ranks (I've modified/simplified the rules a bit):

 I guess only Chinese people can do this challenge!  Okay, fine, here are some English explanations.

0: 4 fours and 2 ones.
1: 6 fours.
2: 6 ones.
3: 6 of any kind except fours and ones.
4: 5 fours.
5: 5 of any kind except for fours.
6: 4 fours.
7: Straight. (1-6)
8: 3 fours.
9: 4 of any kind except 4.
10: 2 fours.
11: 1 four.
12: Nothing.

Input
6 numbers, an array of 6 numbers, or a string of 6 numbers that represent the values of the 6 dice rolls from 1-6
Output
Your program/function may return/output anything to indicate the rank, as long as each rank is indicated by one output and vice versa. Ex. Using the numbers 0-12, 1-13, etc.
Examples(Using 0-12 as the outputs)
[1,1,1,1,1,1]->2
[1,4,4,4,1,4]->0
[3,6,5,1,4,2]->7
[1,2,3,5,6,6]->12
[3,6,3,3,3,3]->5
[4,5,5,5,5,5]->5

This is code-golf, so shortest byte count wins!

(I would've put this in the sandbox, but I wanted the timing to be right. I tried to be as thorough as possible, please let me know if there are any clarifications needed.)

@Shaggy So the OP says that output 0-12 or 1-13 instead

@Shaggy As I stated in the question, the output does not necessarily have to correspond with the label number. The skipped number and random gaps in the image are to simplify the rules a bit - there really aren't definite rules for this tradition, this is just my interpretation.

Shouldn't it be [1,2,3,5,6,6]->13 ??

@J.Doe The examples/test-cases use the indices as result instead of the values. Unlike the values, the 10 isn't skipped.

Should 9 be 四进 4 any (expect 4), 带 plus, 一秀 1 four? Not only 4 any (expect 4)? Some simple Google search also prove this idea: douban (But it's too late to edit now

I took the liberty to edit the picture with ranks numbered from 0 through 12 to avoid the confusion.

@tsh I did say I simplified the rules a bit, and these are the rules that my family plays with. There are no rules that are set in stone, so this is just one interpretation...

Charcoal, 55 bytes
≡⌈Ｅθ№θι⁶Ｉ⁻²⌕14§θχ⁵Ｉ⁻⁵⁼⁵№θ4⁴⎇⁼№θ4⁴§60⁼№θ1²9¹7Ｉ⁻¹²⌊⊘Ｘ²№θ4

Try it online! Link is to verbose version of code. Does not skip 10. Explanation:
≡⌈Ｅθ№θι

Calculate the highest frequency of any digit.
⁶Ｉ⁻²⌕14§θχ

If there is a 6 of a kind, then subtract the position of the digit in the string 14 from 2. This results in 1 for 6 4s, 2 for 6 1s, and 3 for 6 of anything else.
⁵Ｉ⁻⁵⁼⁵№θ4

If there is a 5 of a kind, then the result is 5 unless there are 5 4s, in which case 1 is subtracted.
⁴⎇⁼№θ4⁴§60⁼№θ1²9

If there is a 4 of a kind, then if there are 4 4s then the result is 6 unless there are 2 1s in which case the result is 0, otherwise the result is 9.
¹7

If all the digits are different then the result is 7.
Ｉ⁻¹²⌊⊘Ｘ²№θ4

Otherwise the result is 12 - (4 >> (3 - # of 4s)).

It seems to be best practice to accept the shortest answer, so that's what I'll do :) Unfortunately not many people saw and upvoted your answer...

@Quintec That's not a problem; people are supposed to upvote answers based on the ingenuity of the algorithm or other factor that makes them appreciate the answer.

JavaScript (ES6), 88 bytes
a=>a.map(o=n=>[x=o[n]=-~o[n],6,,,21,9,8^o[1]][a=x<a?a:x])[5]|[,1,4,5,o[1]&2|8,2,4][o[4]]

Try it online!
or Test all possible rolls!
Outputs an integer according to the following mapping:
 Rank | Output       Rank | Output
------+--------     ------+--------
   0  |  31            7  |   7
   1  |  12            8  |   5
   2  |  14            9  |  21
   3  |   8           10  |   4
   4  |  11           11  |   1
   5  |   9           12  |   0
   6  |  29

How?
Method
The output is computed by performing a bitwise OR between:

a bitmask based on \$F\$: the number of 4's
a bitmask based on \$M\$: the maximum number of occurrences of the same dice

Exceptions:

When \$F=4\$: we use a special bitmask \$4b\$ when the two other dice are 1's, or a default bitmask \$4a\$ otherwise.
When \$M=6\$: we use a special bitmask \$6b\$ when we have a six-of-a-kind of 1's, or a default bitmask \$6a\$ otherwise.

Table
Valid combinations of \$F\$ and \$M\$ are highlighted in bold and blue in the following table.
$$\begin{array}{c|c|cccccccc}
&F&0&1&2&3&4a&4b&5&6\\
\hline
M&\text{OR}&0&1&4&5&8&10&2&4\\
\hline
1&6&\color{grey}{6}&\color{blue}{\textbf{7}}&\color{grey}{6}&\color{grey}{7}&\color{grey}{14}&\color{grey}{14}&\color{grey}{6}&\color{grey}{6}\\
2&0&\color{blue}{\textbf{0}}&\color{blue}{\textbf{1}}&\color{blue}{\textbf{4}}&\color{grey}{5}&\color{grey}{8}&\color{grey}{10}&\color{grey}{2}&\color{grey}{4}\\
3&0&\color{blue}{\textbf{0}}&\color{blue}{\textbf{1}}&\color{blue}{\textbf{4}}&\color{blue}{\textbf{5}}&\color{grey}{8}&\color{grey}{10}&\color{grey}{2}&\color{grey}{4}\\
4&21&\color{blue}{\textbf{21}}&\color{blue}{\textbf{21}}&\color{blue}{\textbf{21}}&\color{grey}{21}&\color{blue}{\textbf{29}}&\color{blue}{\textbf{31}}&\color{grey}{23}&\color{grey}{21}\\
5&9&\color{blue}{\textbf{9}}&\color{blue}{\textbf{9}}&\color{grey}{13}&\color{grey}{13}&\color{grey}{9}&\color{grey}{11}&\color{blue}{\textbf{11}}&\color{grey}{13}\\
6a&8&\color{blue}{\textbf{8}}&\color{grey}{9}&\color{grey}{12}&\color{grey}{13}&\color{grey}{8}&\color{grey}{10}&\color{grey}{10}&\color{blue}{\textbf{12}}\\
6b&14&\color{blue}{\textbf{14}}&\color{grey}{15}&\color{grey}{14}&\color{grey}{15}&\color{grey}{14}&\color{grey}{14}&\color{grey}{14}&\color{grey}{14}\\
\end{array}$$
All other combinations (in gray) can't possibly happen. For instance, if we have three 4's, we must have \$M\ge3\$. But because there are only 3 other remaining dice in such a roll, we also have \$M\le3\$. So, there's only one possible value of \$M\$ for \$F=3\$.
Example
If we have a straight, each dice appears exactly once. So we have \$M=1\$ and \$F=1\$. Using the bitmasks described in the above table, this leads to:
$$6 \text{ OR } 1 = 7$$
There is another \$7\$ in the table, but it's invalid. Therefore, a straight is uniquely identified by \$7\$.
Commented
a =>                   // a[] = input array, reused as an integer to keep track of the
  a.map(               //       maximum number of occurrences of the same dice (M)
    o =                // o = object used to count the number of occurrences of each dice
    n =>               // for each dice n in a[]:
    [                  //   this is the lookup array for M-bitmasks:
      x =              //     x = o[n] = number of occurrences of the current dice
        o[n] = -~o[n], //     increment o[n] (we can't have M = 0, so this slot is not used)
      6,               //     M = 1 -> bitmask = 6
      ,                //     M = 2 -> bitmask = 0
      ,                //     M = 3 -> bitmask = 0
      21,              //     M = 4 -> bitmask = 21
      9,               //     M = 5 -> bitmask = 9
      8 ^ o[1]         //     M = 6 -> bitmask = 14 for six 1's, or 8 otherwise
    ][a =              //   take the entry corresponding to M (stored in a)
        x < a ? a : x] //   update a to max(a, x)
  )[5]                 // end of map(); keep the last value
  |                    // do a bitwise OR with the second bitmask
  [                    // this is the lookup array for F-bitmasks:
    ,                  //   F = 0 -> bitmask = 0
    1,                 //   F = 1 -> bitmask = 1
    4,                 //   F = 2 -> bitmask = 4
    5,                 //   F = 3 -> bitmask = 5
    o[1] & 2 | 8,      //   F = 4 -> bitmask = 10 if we also have two 1's, 8 otherwise
    2,                 //   F = 5 -> bitmask = 2
    4                  //   F = 6 -> bitmask = 4
  ][o[4]]              // take the entry corresponding to F (the number of 4's)

R, 100 bytes
Encode the score as a bunch of indexed conditionals. Simpler than my first stringy-regex approach.
Edit- fixed bug and now rank all rolls.
function(d,s=sum(d<2))min(2[s>5],c(11,10,8,6-6*!s-2,4,1)[sum(d==4)],c(7,12,12,9,5,3)[max(table(d))])

Try it online!

JavaScript (Node.js), 169 bytes
a=>-~"114444|123456".search(w=a.sort().join``,[l,t]=/(.)\1{3,}/.exec(w)||[0],l=l.length)||(l>5?3-"14".search(t):l>3?4+(5.5-l)*(t-4?4:2):[,13,12,11,9][w.split`4`.length])

Try it online!
Returns 1..13

Python 2,  148  119 bytes
-27 bytes thanks to ovs's (1. use of .count allowing a map to be used; 2. removal of redundant 0 in slice; 3. use of an in rather than a max; 4. shortened (F==4)*O==2 to F==4>O==2 [since golfed to F>3>O>1])
C=map(input().count,range(1,7))
O,F=C[::3]
print[F>3>O>1,F>5,O>5,6in C,F>4,5in C,F>3,all(C),F>2,4in C,F>1,F,1].index(1)

Try it online!

@ovs oh heh .count >_< nice

One last suggestion: As d is only needed one, this is shorter as a full program.

Python 3 version: 131 bytes

@ovs - thanks for all your golfing. Saved another two to boot.

Gotta love python conditional chaining! Also, the spec now doesn't skip the number 10

Pyth, 60 bytes
eS[@[ZhZ2 4*6hq2hJuXtHG1Qm0Q8hT)@J3*5hSJ*Tq6hJ*<3KeSJ@j937TK

Maps to reversed rank, 0-12. Try it online here, or verify all the test cases at once here.
The full mapping used is as follows:
12: 4 fours and 2 ones.
11: 6 fours.
10: 6 ones.
 9: 6 of any kind except fours and ones.
 8: 5 fours.
 7: 5 of any kind except for fours.
 6: 4 fours.
 5: Straight. (1-6)
 4: 3 fours.
 3: 4 of any kind except 4.
 2: 2 fours.
 1: 1 four.
 0: Nothing.

This works by mapping the dice values to frequencies, then calculating the value of several rules, and taking the maximum of the set.
Implicit: Q=eval(input()), Z=0, T=10

JuXtHG1Qm0Q   Input mapping
 u     Q      Reduce each element of the input, as H, ...
        m0Q   ...with initial value, G, as an array of 6 zeroes (map each roll to 0)
   tH           Decrement the dice roll
  X  G1         Add 1 to the frequency at that point
J             Store the result in J

@[ZhZ2 4*6hq2hJuXtHG1Qm0Q8hT)@J3   Rule 1 - Values for sets including 4
  Z                               *0
   hZ                             *1 (increment 0)
     2                            *2
       4                          *4
              JuXtHG1Qm0Q          Input mapping (as above)
             h                     First value of the above - i.e. number of 1's
           q2                      1 if the above is equal to 2, 0 otherwise
        *6h                       *Increment and multiply by 6
                                   Maps to 12 if there are 2 1's, 6 otherwise
                         8        *8
                          hT      *11 (increment 10)
 [                          )      Wrap the 7 starred results in an array
                             @J3   Get the 4th value of the input mapping - i.e. number of 4's
@                                  Get the value at that position in the array

*5hSJ   Rule 2 - Straight
  hSJ   Smallest frequency in input mapping (first, sort, J)
        For a straight, smallest value will be 1, 0 otherwise
*5      Multiply by 5

*Tq6hJ   Rule 3 - 6 1's
    hJ   Frequency of 1's (first value from input mapping)
  q6     1 if the above == 6, 0 otherwise
*T       Multiply by 10

*<3KeSJ@j937TK   Rule 4 - 4,5,6 of a kind, other than 4's
   KeSJ          Get the max frequency from input mapping, store in K
        j937T    [9,3,7]
       @     K   Get the element at K'th position in the above (modular indexing)
 <3K             1 if 3<K, 0 otherwise
*                Multiply the previous 2 results

eS[   Wrapping it all up!
  [   Wrap all the above rules in an array
eS    Take the max value of the above, implicit print

Retina, 137 126 bytes
4
A
O`.
^11A+
0
1+$
2
^A+
1
(.)\1{5}
3
^.A+
4
.?(.)\1{4}.?
5
^..A+
6
12356A
7
.+AAA$
8
.*(.)\1{3}.*
9
.+AA$
10
.+A$
11
.{6}
12

-11 bytes thanks to @Neil.
Output is 0-indexed (0..12).
Try it online or verify all test cases.
Explanation:
Replace every 4 with an 'A':
4
A

Sort all the input-digits (the A's will be at the back):
O`.

Every other two lines replaces the input with the expected output:
Regex         Replacement  Explanation

^11A+         0            starts with "11", followed by any amount of A's
1+$           2            ends with any amount of 1s
^A+           1            starts with any amount of A's
(.)\1{5}      3            six of the same characters
^.A+          4            starts with any character, followed by any amount of A's
.?(.)\1{4}.?  5            five of the same characters,
                           with optionally a leading or trailing character
^..A+         6            starts with any two characters, followed by any amount of A's
12356A        7            literal "12356A" match
.+AAA$        8            any amount of leading characters, ending with three A's
.*(.)\1{3}.*  9            four of the same characters,
                           with optionally any amount of leading/trailing chars
.+AA$         10           any amount of leading characters, ending with two A's
.+A$          11           any amount of leading characters, ending with a A
.{6}          12           any six characters

Very clever, but I think you can go one step further and replace 4 with something outside of the range 1-6 to get it to sort to one end automatically (not sure whether it makes a difference which end you sort to).

@Neil Ah, that's clever as well! Thanks.

05AB1E, 57 55 bytes
4¢4@U.M¢©5-Di14¹нk2αë>Di5X-ë>i9X3*-¹1¢2Ê*ë®i7ë¹4¢o;ï12α

Port of @Neil's Charcoal answer, because my initial approach was already at 60 bytes and I wasn't done yet. My current answer can probably be golfed some more, though.
Input as a list of digits.
Try it online or verify all test cases.
Explanation:
4¢              # Count the amount of 4s in the (implicit) input-list
  4@            # Check if it's larger than or equal to 4
                # (results in 1 for truthy; 0 for falsey)
    U           # Pop and store that result in variable `X`
.M              # Push the most frequent number in the (implicit) input-list
  ¢             # Pop and push the count of that number in the (implicit) input-list
   ©            # Store it in the register (without popping)
5-Di            # If the maximum count - 5 is exactly 1 (so the maximum count is 6):
    14          #  Push 14
      ¹н        #  Push the first digit of the input-list
        k       #  Get its index in 14, resulting in -1, 0, or 1
         2α     #  Take the absolute difference with 2
                #  (This covers cases 1, 2, and 3)
ë>Di            # Else-if the maximum count - 5 + 1 is exactly 1 (so the maximum count is 5):
    5           #  Push 5
     X-         #  And subtract variable `X`
                #  (This covers cases 4 and 5)
ë>i             # Else-if the maximum count - 5 + 2 is exactly 1 (so the maximum count is 4):
   9            #  Push 9
    X3*         #  Multiply variable `X` by 3
       -        #  And subtract it from the 9
        ¹1¢     #  Count the amount of 1s in the input-list
           2Ê   #  Check if it's NOT equal to 2 (1 if truthy; 0 if falsey)
             *  #  Multiply it with the 9 or 6
                #  (This covers cases 0, 6, and 9)
ë®i             # Else-if the maximum count is 1:
   7            #  Push a 7
                #  (This covers case 7)
ë               # Else (maximum count is 2 or 3):
 ¹4¢            #  Count the amount of 4s in the input-list
    o           #  Take 2 to the power this count
     ;ï         #  Halve and floor it
       12α      #  And take the absolute difference with 12
                #  (This covers cases 8, 10, 11, and 12)
                # (Output the top of the stack implicitly as result)

Ruby, 100 bytes
->a{%w(^2.*4$ 6$ ^6 6 5$ 5 4$ ^1*$ 3$ 4 2$ 1$ .).index{|b|/#{b}/=~([1,*a,4].map{|c|a.count(c)}*'')}}

Try it online!
How it works:
Count occurrences of every number in the array, prepend number of 1s and append number of 4s.
After that, try to match with different regex patterns.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
True color code
True color (24-bit) at Wikipedia is described in pertinent part as

24 bits almost always uses 8 bits of each of R, G, B. As of 2018
  24-bit color depth is used by virtually every computer and phone
  display and the vast majority of image storage formats. Almost all
  cases where there are 32 bits per pixel mean that 24 are used for the
  color, and the remaining 8 are the alpha channel or unused.
224 gives 16,777,216 color variations. The human eye can discriminate up to ten million colors[10] and since the gamut of a display is smaller than the range of human vision, this means this should cover that range with more detail than can be perceived. ...
...
Macintosh systems refer to 24-bit color as "millions of colors". The term "True color" is sometime used to mean what this article is calling "Direct color".[13] It is also often used to refer to all color depths greater or equal to 24.

All 16,777,216 colors

Task
Write a program which generates and returns all 16,777,216 color variations within an array as strings in the CSS rgb() function

5.1. The RGB functions: rgb() and rgba()
The rgb() function defines an RGB color by specifying the red,
  green, and blue channels directly. Its syntax is:
rgb() = rgb( <percentage>{3} [ / <alpha-value> ]? ) |
        rgb( <number>{3} [ / <alpha-value> ]? )
<alpha-value> = <number> | <percentage>

The first three arguments specify the red, green, and blue channels of
  the color, respectively. 0% represents the minimum value for that
  color channel in the sRGB gamut, and 100% represents the maximum
  value. A <number> is equivalent to a <percentage>, but with a
  different range: 0 again represents the minimum value for the color
  channel, but 255 represents the maximum. These values come from the
  fact that many graphics engines store the color channels internally as
  a single byte, which can hold integers between 0 and 255.
  Implementations should honor the precision of the channel as authored
  or calculated wherever possible. If this is not possible, the channel
  should be rounded to the closest value at the highest precision used,
  rounding up if two values are equally close.
The final argument, the <alpha-value>, specifies the alpha of the
  color. If given as a <number>, the useful range of the value is 0
  (representing a fully transparent color) to 1 (representing a fully
  opaque color). If given as a , 0% represents a fully
  transparent color, while 100% represents a fully opaque color. If
  omitted, it defaults to 100%.
Values outside these ranges are not invalid, but are clamped to the
  ranges defined here at computed-value time.
For legacy reasons, rgb() also supports an alternate syntax that
  separates all of its arguments with commas:
rgb() = rgb( <percentage>#{3} , <alpha-value>? ) |
        rgb( <number>#{3} , <alpha-value>? )

Also for legacy reasons, an rgba() function also exists, with an
  identical grammar and behavior to rgb().

or RGB hexadecimal notation #RRGGBB format

5.2. The RGB hexadecimal notations: #RRGGBB
The CSS hex color notation allows a color to be specified by
  giving the channels as hexadecimal numbers, which is similar to how
  colors are often written directly in computer code. It’s also shorter
  than writing the same color out in rgb() notation.
The syntax of a <hex-color> is a <hash-token> token whose value
  consists of 3, 4, 6, or 8 hexadecimal digits. In other words, a hex
  color is written as a hash character, "#", followed by some number of
  digits 0-9 or letters a-f (the case of the letters doesn’t matter -
  #00ff00 is identical to #00FF00).
The number of hex digits given determines how to decode the hex
  notation into an RGB color:
6 digits
The first pair of digits, interpreted as a hexadecimal number, specifies the red channel of the color, where 00 represents the
  minimum value and ff (255 in decimal) represents the maximum. The
  next pair of digits, interpreted in the same way, specifies the green
  channel, and the last pair specifies the blue. The alpha channel of
  the color is fully opaque.

EXAMPLE 2 In other words, #00ff00 represents the same color as rgb(0 255 0) (a lime green).

See Editor's Draft of CSS Color Module Level 4
Examples
CSS rgb() function (space character can be substituted for comma character, e.g., rgb(0 255 0))
// `rgb()` `<percentage>` as strings in resulting array
['rgb(0%,0%,0%)', ...,'rgb(0%,255%,0)', ...'rgb(255,255,255)']

// `rgb()` `<number>` as strings in resulting array
['rgb(0,0,0)', ...,'rgb(0,255,0)', ...'rgb(255,255,255)']

CSS RGB hexadecimal notation RRGGBB
// RGB hexadecimal notation as strings in resulting array
['#000000', ...,'#00ff00', ...'#ffffff']

Winning criteria
Least bytes used to write the program.

Comments are not for extended discussion; this conversation has been moved to chat.

R, 25 bytes
sprintf("#%06X",1:2^24-1)

Try it online!

Not familiar with R, but does this fail to output #000000?

No, it outputs #000000. See the TIO link

@jimbo1qaz a:b-c makes a vector from a-c to b-c, inclusive (it makes a vector from a to b, then subtracts c from each entry).

Ahh, so R specifies colons with tighter precedence than arithmetic... Unlike Python slices.

Python 2, 77 40 39 37 bytes
print['#%06X'%c for c in range(8**8)]

Try it online!
-1  byte thanks to Digital Trauma
-2  bytes thanks to dylnan

How can we output the entire result of F[0:16777216] at TIO The output exceeded 128KiB?

@guest271314 TIO's output size is limited. That's why TFeld added the F[:10] to see the first 10 items of the list, and F[-10:] to see the last ten items of the list. And he also proved it contained all 16,777,216 items by outputting the length of the result-list.

@guest271314 If you really want all the output, run it locally

Why do you need the lambda? The list comprehension itself is a valid answer, isn't it?

Try 8**8 instead of 1<<24 to save a byte

how about print['...

@Adirio A value alone does not count as an implementation of a challenge.

@dylnan I had print, but the challenge says to return the colors, so I wasn't sure if it was valid.

A value alone in a Python terminal prints itself when not assigned to a variable, shouldn't that be equivalent to the current solution? You would still need to assign it to a variable in TIO and print some partials as you did, but if printing to the console is a valid answer i do not see why the value itself specifying that is in a Python console is different from an actual print.

@Adirio You yourself wrote [...] in a Python terminal [...] -- thus your submission would be written in Python 2 REPL, not Python 2.

Oh, Python 2 REPL is considered separately, got it. I'm new to code golf as you can see.

JavaScript (ES7), 65 62 61 bytes
Saved 3 4 bytes thanks to @tsh
Returns an array of #RRGGBB strings.
_=>[...Array(n=8**8)].map(_=>'#'+(n++).toString(16).slice(1))

Try it online!
(truncated output)

Current Python solution use 8**8 instead of 1<<24. It would work on JS too.

PowerShell, 28 26 bytes
1..16mb|%{"#{0:x6}"-f--$_}

Try it online!
Loops from 1 to 16mb (16777216). Each iteration, we use the -format operator acting on the current number pre-decremented --$_ against the string "#{0:x6}". Here, we're specifying hex values, padded to 6 digits, with a hash # in front. On TIO, limited to 60 seconds / 128KiB of output. Change the 1 to (16mb-5) to see how it ends.

Common Lisp, 42 bytes
(dotimes(i 16777216)(format t"#~6,'0x "i))

Try it online!

May be the first format-golf I've seen. +1 on that basis alone.

Japt, 14 bytes
Outputs as #rrggbb.
G²³ÇsG ùT6 i'#

Try it (Limited to the first 4096 elements)

Explanation
G                  :16
 ²                 :Squared
  ³                :Cubed
   Ç               :Map the range [0,result)
    sG             :  Convert to base-16 string
       ù           :  Left pad
        T          :   With 0
         6         :   To length 6
           i'#     :  Prepend "#"

Interesting language. Any idea why the letter T is used for "0"? I get 16 -> G to save a byte, but T -> 0 doesn't accomplish the same.

@Alec Because if you replace T with 0, it joins with the 6 and becomes 06.

Ah, gotcha. So is there one letter per digit for cases where you don't want it to join with the previous/next digit?

@Alec, as geokavel said, in this particular case, it saves me a byte in having to use a comma to delimit the 2 arguments being passed to ù. Another typical use case for it is to use it as a counter when you need to increment a variable while, for example, mapping over an array. And, of course, as it's a variable, you can simply assign a value to it too, if needed. 0 is the only single digit integer that has it's own variable, though - well, technically, 7 as the 6 input variable U-Z default to 0. The other integers assigned to variables in Japt are: -1, 10-16, 32, 64 & 100.

If you'd like to learn more about Japt, feel free to ping me in our chatroom.

14 bytes but 8008 elements here

Haskell, 41 bytes
l="ABCDEF"
mapM id$"#":(['0'..'9']++l<$l)

Try it online!

05AB1E, 15 14 10 bytes
15Ýh6ãJ'#ì

Try it online.
Explanation:
15Ý           # Create a list in the range [0, 15]
   h          # Convert each to a hexadecimal value
    6ã        # Create each possible sextuple combination of the list
      J       # Join them together to a single string
       '#ì    # And prepend a "#" before each of them

Batch, 87 bytes
@set s= in (0,1,255)do @
@for /l %%r%s%for /l %%g%s%for /l %%b%s%echo rgb(%%r,%%g,%%b)

Outputs in CSS format. The variable substitution happens before the for statement is parsed so the the actual code is as follows:
@for /l %%r in (0,1,255)do @for /l %%g in (0,1,255)do @for /l %%b in (0,1,255)do @echo rgb(%%r,%%g,%%b)

C# (Visual C# Interactive Compiler), 47 bytes
Enumerable.Range(0,1<<24).Select(q=>$"#{q:X6}")

Try it online!

MATL, 17 15 bytes
10W:q'#%06X,'YD

Try it online!
The TIO version displays the first 2^10 only as not to time out. I included the final iteration in the footer to show that it indeed terminates at  #FFFFFF. Saved one byte by changing to fprintf instead of manually assembling the string. Outputs a comma-separated list.
Explanation
24W:q            % Range from 0 to 2^24-1
     '#%06X,'    % fprintf format spec (# followed by hexadecimal, zero-padded, fixed-width, followed by newline)
             YD  % Call fprintf. Internally loops over range.

C# (.NET Core), 75 bytes
()=>{int i=1<<24;var a=new string[i];for(;i-->0;)a[i]=$"#{i:X6}";return a;}

Try it online!
Port of JAVA 10 version with C# interpolated string format

K (oK), 19 bytes
Solution:
$(3#256)\'!16777216

Try it online! (limited to first 500 numbers)
Explanation:
Dump out rgb strings. Convert each number between 0 and 16777216 to base 256, then convert to strings...
$(3#256)\'!16777216 / the solution
          !16777216 / range 0..16777215
 (     )\'          / split each both
  3#256             / 256 256 256
$                   / string

APL (Dyalog Unicode), 47 43 20 bytes
'#',(⎕D,⎕A)[↑,⍳6⍴16]

Try it online!
Given enough time/memory, this anonymous function will output all \$2^{24}-1\$ color codes. To see this, you can swap the 6⍴ for a 4⍴ in the code, and you'll see it output every code with up to 4 digits.
Thanks to @Dzaima and @ngn for the 23 bytes.
Uses ⎕IO←0.
How:
'#',(⎕D,⎕A)[↑,⍳6⍴16] ⍝ Main function
               ⍳6⍴16  ⍝ Generate every possible 6 digit hex number in a matrix format
              ,       ⍝ Ravel the matrix (from a 16x16x16x16x16x16 matrix to a 16^6x2 list)
             ↑        ⍝ Mix; (turns the list into a 16^6x2 matrix)
    (⎕D,⎕A)[       ] ⍝ Use that matrix to index the vector of the digits 0-9 concatenated with the alphabet.
'#',                  ⍝ Then prepend a '#' to each.

Ruby, 31 bytes
$><<("#%06x\n"*d=2**24)%[*0..d]

Try it online!

I’m kind of amazed % takes a string that long and an array that long. FYI you can save a byte by using a literal line break instead of \n.

Batch, 69 + 4 = 73
g.cmd, 69
for /L %%A in (0,1,16777215)do cmd/kexit %%A&set #%%A=#!=exitcode:~2!

Saves the hexadecimal value with form #RRGGBB into an 'Array'.
g.cmd is to be called using cmd /V/Q/K g.cmd. This is where the + 4 comes from, /V/Q, counting as 4 additional characters compared to just cmd /K g.cmd. This sets up an environment that has the 'Array' in memory.  It also takes forever to run, so use very low values to try or break execution using Ctrl+C

Breakdown
Invokation

/V enables delayed expansion, but is shorter than setlocal EnableDelayedExpansion, which is why we need the cmd call in the first place
/Q omits the output and is equivalent to @echo off
/K lets you execute an expression (In this case g.cmd) and does not exit afterwards, so you can check the 'Array' by using set #

g.cmd
for /L %%A IN (0,1,16777215) DO (
    cmd /k exit %%A
    set #%%A=#!=exitcode:~2!
)

This bit uses a trick documented here to convert a normal number to a hexadecimal, then saves that value into an 'Array'.

I've been calling that storing structure an 'Array' but that is not actually right as true Arrays do not exist in Batch. BUT you can name variables so that they have arraylike names, like so:
set elem[1]=First element
set elem[2]=Second one

or, like in this case:
set #1=First element
set #2=Second one

You can still access them via !#%position%!

I'm not sure. But maybe /V/Q/K may be claimed as "arguments" for interpreter and count as "3 + 69 = 72". meta

I didn't know that and will update my answer accordingly. Thx @tsh

V, 25 bytes
8É00lrx16777215ñÄ<C-a>ñ0<C-v>Gls#

Try it online! (replaced 16777215 by 31)
Explanation
8É0                                " insert 8 zeroes
   0l                              " move cursor to the second character
     rx                            " replace by x
       16777215ñ      ñ            " 16777215 times do ..
                Ä                  " .. duplicate line
                 <C-a>             " .. increment (leading 0x makes sure it uses hexadecimals)
                       0<C-v>      " move cursor to beginning of line and start selection
                             Gl    " select the column with 0x
                               s#  " replace by #

Java 10, 87 84 bytes
v->{int i=1<<24;var r=new String[i];for(;i-->0;)r[i]="".format("#%06X",i);return r;}

-3 bytes thanks to @archangel.mjj.
Try it online (limited to the first 4,096 items).
Explanation:
v->{                       // Method with empty unused parameter & String-array return-type
  int i=1<<24;             //  Integer `i`, starting at 16,777,216
  var r=new String[i];     //  Result String-array of that size
  for(;i-->0;)             //  Loop `i` in the range (16777216, 0]
    r[i]=                  //   Set the `i`'th item in the array to:
      "".format("#%06X",i);//   `i` converted to a hexadecimal value (of size 6)
  return r;}               //  Return the result-array

Ah, you posted this while I was writing my post, so we have very similar answers. You can improve by three bytes with r[i]="".format("#%06X",i);

@archangel.mjj Ah, I'm an idiot. Thanks! I actually had "".format("#%06X",i) before since I saw it in the Python answer, but I dropped the answer because I couldn't get it to work fast enough for TIO. Then I saw everyone just outputting the first 4,096 items on TIO, so I wrote the answer again, forgetting about "#%06X"... >.<

@KevinCruijssen I never knew you could do var r in Java..

@FireCubez It's new since Java 10. :) Here a relevant tip to see what is and isn't possible with var in Java.

Groovy, 53 bytes
c={a=[];(1<<24).times{a.add "".format("#%06x",it)};a}

Function definition. c() returns an ArrayList (I assume that's fine, even through the question asks for an array).
Ungolfed, with implicit types:
ArrayList<String> c = {
    ArrayList<String> a = []
    (1 << 24).times {
        a.add("".format("#%06x", it)) // add the hex-formatted number to the list.
    }
    return a
}

Try it online!

PHP, 68 62 bytes
This is supposed to be placed inside a file, the array is returned in the end, to be usable.
<?foreach(range(0,1<<24)as$i)$a[]=sprintf('#%06x',$i);return$a;

To have access to the array, simply give the result of the include (e.g.: $a = include 'xyz.php';) to a variable.

Thanks to @manatwork for saving me 6 bytes and fix a goof.

Are you sure this will ever output hex digits with the %1$06d format string? And I see no reason for using 1$. You could reduce the length by including the “#” in the format string: #%06x. Which would come handy when adding extra characters to fix the range, as currently counts up to 16777216 = #1000000.

Well, it would .... If I didn't forgot to change %d to %x. And completely forgot about moving the # inside the sprintf() call. You saved me 6 bytes. Thank you

Lua, 47 45 bytes
for i=1,8^8 do s='#%06X'print(s:format(i))end

Try it online!

You could toss your variable s and save a byte.

MATL, 11 bytes
'#'5Y26Z^Yc

Try it online! (with only three hex digits instead of six)
Explanation
'#'   % Push this character
5Y2   % Push '01234567890ABCDEF'
6     % Push 6
Z^    % Cartesian power. Gives a (16^6)×6 char matrix
Yc    % String concatenation. '#' is implicitly replicated
      % Implicitly display

Bash + jot, 22
jot -w\#%06X $[8**8] 0

Try it online!

Perl 5, 31 bytes
printf"#%06X
",$_ for 0..8**8-1

Try it online!

T-SQL, 122 117 bytes
Returns a 16,777,216-row table of #RRGGBB strings. The line break is for readability only:
WITH t AS(SELECT 0n UNION ALL SELECT n+1FROM t WHERE n<16777215)
SELECT'#'+FORMAT(n,'X6')FROM t option(maxrecursion 0)

Uses a recursive CTE for a number table from 0 to 2^24-1, then uses the built-in FORMAT command (available in SQL 2012 or later) to turn it into a 6-digit hex string. Attach the # to the front, and we're done.
Edit 1: Removed POWER() function, the number was shorter :P

Jelly, 12 bytes
⁴ṖṃØHœċ6ṭ€”#

Try it online!

Pascal (FPC), 67 65 bytes
Saved 2 bytes thanks to @tsh
var i:int32;begin for i:=1to 1<<24do writeln('#',hexStr(i,6))end.

Try it online!
Writes on standard output. It can be stored in an array using procedure, but it will be longer.

Loop from 1 to 1<<24 would save 2 bytes, as hexStr(1<<24,6) would be 000000.

@tsh Now it looks obvious, thanks!

Jelly, 8 bytes
ØHṗ6”#;Ɱ

Try it online! (note: uses 2 rather than 6 as 6 times out on TIO)
Function submission (because Jelly full programs will, by default, print lists of strings with no delimiters between them, making it hard to see the boundaries). The TIO link contains a wrapper to print a list of strings using newlines to separate them.
Explanation
ØHṗ6”#;Ɱ
ØH         All hex digits (“0123456789ABCDEF”)
  ṗ6       Find all strings of 6 of them (order relevant, repeats allowed)
    ”#;    Prepend “#”
       Ɱ     to each of the resulting strings

Out of interest -- why did you make your answer a community wiki?

@JonathanFrech: I do this for all my posts because a) it reduces the incentive for people to game the reputation system (as the post doesn't give reputation), b) I'm happy to have my posts edited and the community-wiki marker is a way to indicate that. Stack Exchange's reputation system is more or less completely broken: on a past account, I once intentionally rep-capped every day for a week to show how easy the system was to game. Nowadays I pretty much want no part in it, especially as higher reputation simply just makes the site try to persuade you to moderate it.

Just curious -- on which stack did you achieve to game the reputation system?

@JonathanFrech: This one. I was a 20k user, but eventually deleted my account because it was kind-of messing up my life, and because the reputation system was actively pushing me into making content that made the site worse, as opposed to better.

PHP, 43 bytes
<?php for(;$i<1<<24;printf("#%06X ",$i++));

Try it online!

1<<24 --> 8**8 should work here to, credits to @DigitalTrauma

Perl 6, 26 bytes
{map *.fmt("#%06X"),^8**8}

Try it online!
Uses the same format as everyone else. Times out on TIO.
Or, in rgb format:
31 bytes
{map {"rgb($_)"},[X] ^256 xx 3}

Try it online!

I think the rgb output should be rgb(0, 0, 0) including the string rgb.

@nwellnhof Updated (though it ended up shorter to do hexadecimal)

><>, 99 bytes
I added an extra space here to fix the layout, because of the က character, which is 4096 or sqrt(2^24) which is used in the main loop invariant.
!/ i&"ÿĀ"::*3ep4ep5ep\1+:&:5eg%n$o:4eg,:1%-5eg%n$o3eg,:1%-5eg%noo:*&:&(?;!
 >"က"a"),,(bgr"oooo&\

The idea is pretty simple: use the register as a counter n and output rgb(n%255, (n/255)%255, (n/255^2)%255).
Pretty sure some bytes can be golfed off, but it's a start.
Try it online!

Tcl, 53 bytes
time {puts #[format %06X [expr [incr i]-1]]} 16777216

Try it online!
Times out on TIO, but runs fine on my machine!

Scala, 40 bytes
(1 to 1<<24).map(x=>println(f"#$x%06x"))

Try it online!

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Recursively count lines of code, excluding empty lines and comments
Requirement: Write a program (in any language) that counts the number of lines of code in files matching *.sh in the directory tree starting from the directory that the program is executed in, excluding lines that are empty, only contain whitespace, or are just single-line comments (a line which the first character that is not a whitespace character is #). Files will only contain printable ASCII characters.
Output: A single integer representing the total count (can be followed by a newline too).
Here is an expanded solution in Python:
import os, sys

def recurse(dir = './'):
    count = 0

    for file in os.listdir(dir):
        if not os.path.isfile(dir + file):
            count += recurse(dir + file + '/')
        elif file.endswith('.sh'):
            with open(dir + file, 'r') as f:
                for line in f.read().split('\n'):
                    if (not line.strip().startswith('#')) and (not line.strip() == ''):
                        count += 1

    return count

sys.stdout.write(recurse())

Files will only contain printable ASCII characters. -- can a file contain a /?

Powershell 3+, 40 byte
(ls *.sh -r|sls '^\s*(#|$)' -a -n).Count

ls *.sh -r gets a file names from the directory tree. sls (alias for Select-String) gets all strings (-a is shortcut for -AllMatches) that not mathces (-n is alias for -NotMatch) to the pattern '^\s*(#|$)'.

Side note: ().Count was introduced in PowerShell 3; in PowerShell 2 you can use |measure instead although that also tries to give you other measurements not relevant here.

I'm agree (....|measure).Count. But Powershell 2 is deprecated since the summer 2017. I'm not sure the version 3+ requires an explicit statement. In any case, I've specified the version. Thanks.

Ah! I keep forgetting about Select-String! Nice solution.

PowerShell, 47 46 bytes
ls *.sh -r|gc|%{$o+=!($_-match'^\s*(#|$)')};$o

Try it online! (will always return 0 since there aren't any files)
Try it online! (here's a link that populates a dummy file so you can see the process)
-1 byte thanks to Neil
ls is alias for Get-ChildItem, specifying *.sh with the -recurse parameter, then we get-content of files. For each of those lines |%{...}, we accumulate into our $output a one if the Boolean !(...) statement is truthy. Inside the statement is a simple regex -match against the whitespace-only/comment/blank lines. Finally we leave $o on the pipeline.
The implicit Write-Output that happens at program completion adds a trailing newline, but that shouldn't matter in this case because the variable $o itself doesn't have a trailing newline nor does the actual return variable. It's a quirk of the shell, not a quirk of the program. For example, saving this to a script and executing that script in a pipeline will not have a newline.

You can write $-notmatch'^\s*(#|$)' instead !($_-match'^\s*(#|$)'). The length is same. But expression is more clear.

ls *.sh -r|gc| saves a byte.

Thanks @Neil - nice catch.

Linux Shell, 30 60 bytes
confirmed in dash
cat `find . -name \*.sh`|tr -d " \t"|grep .|grep -v ^#|wc -l

find . -name \*.sh: find files matching the pattern, list with path
cat '': list contents of these files
tr -d " \t": trim tabs and spaces
grep .: remove empty lines
grep -v ^#: remove comments
wc -l: count lines of output

the comment # can be found after spaces.

@mazzy, nope, after tr -d " \t" they can not.

R, 70  64 bytes
sum(!grepl("^(#|$)",unlist(lapply(dir(,".sh$",r=T),readLines))))

Explanation:
The dir function has the recursive flag set.
readLines returns the lines of a file in a vector, which are then flattened with unlist.

I haven't tested it... but can you save bytes by using grepl, adding a ! and getting rid of |1 and of inv=T?

Haskell, 211 210 bytes
import System.Directory
f x=listDirectory x>>=fmap sum.mapM(\d->doesFileExist d>>=(#d)).map((x++"/")++)
p#d|p=do c<-readFile d;pure$sum[1|take 3(reverse d)=="hs.",(q:_):_<-map words$lines c,q/='#']|1<2=f d
f"."

Oh dear, without shell glob and regex you have to do all the work by yourself. Maybe there's somewhere a module for it. Also, IO code in Haskell requires some overhead to get the types right.
f x =                             -- main function, expects a directory 'x'
    listDirectory x >>=           -- read content of directory (without "." and "..")
              map((x++"/")++)     --   for each entry: prepend current directory 'x' and a slash
          mapM(\d->doesFileExist d>>=(#d))
                                  --   for each entry: call function '#' with
                                  --        first parameter: a boolean, True if it's a regular file, False if it's a directory
                                  --        second parameter: the filename itself
                                  --        '#' returns a list of valid lines for each file
     fmap sum                     --   sum this list

p#d
   |p                             -- if 'p' is True (i.e. 'd' is regular file)
     do c<-readFile d;            --   read the content 'c' of file 'd' and
        pure$sum[1|      ]        --   return the number of lines
              take 3(reverse d)=="hs."
                                  --     (only if the file end with .sh)
              (q:_):_<-map words$lines c,q/='#'
                                  --     where the first word doesn't start with a hash sign
                                  --       (function 'words' strips leading whitespace)
   |1<2                           -- else ('d' is a directory)
       =f d                       --   examine d

f"."                              -- start with current directory

Bash + GNU utilities, 55
find . -name \*.sh -exec cat {} +|grep -Evc '^\s*(#|$)'

Try it online!

Does find . -name \*.sh|xargs cat|grep -Evc '^\s*(#|$)' work?

Or even cat **/*.sh|..., though it needs globstar to be turned on (or a different shell such as zsh, csh/tcsh).. Not sure if you'd need to include it in your bytecount.

Batch + Internal tools, 80 76 Bytes
@for /F %%F IN ('findstr/SV "^\s*# ^\s*$" *.sh^|find/C":"')DO @set/P=%%F<NUL

Uses the builtin findstr to fetch the lines, then counts these line using find /C.
This however produces output with a newline, so we need to convert that into an output without. This is done by using for /F to fetch the output and then use <NUL set /P to output without the trailing newline.

Python 2, 149 146 144 140 149 146 116 bytes
import os
print sum(l.strip()[:1]not in'#'for a,b,c in os.walk(".")for n in c for l in open(a+"/"+n)if'.sh'==n[-3:])

Try it online!
Reports 0 on TIO but works locally. Probably not any .sh files in whatever is the current directory.
I think it now works correctly on TIO and fixed a bug for +6, both thanks to @JonathanAllen
-30 with thanks to @ovs
Alternative for 139 bytes but only works on Windows.
import os
os.system('dir/s/b *.sh>f')
print sum(sum((0,1)[x.strip()>''and'#'!=x.strip()[0]]for x in open(l[2:].strip()))for l in open("f"))

Creates a temporary file f to store the results for the dir command.

You can create some files at TIO. Unfortunately that shows up a bug - I believe you'd need to open(os.path.join(a_path_prefix, n)) for it to actually work (although just using + will of course be golfier!))

(0,1)[l.strip()>''and'#'!=l.strip()[0]] can become all([l.strip()>'','#'!=l.strip()[0:]]) to save 1 byte.

Thanks @JonathanAllan. Will delete for now as I don't have time to fix.

@mypetlion - thanks but temporarily deleted due to a bug spotted by JonathanAllan

That's impressive, nice job. I tried to cut off a byte or two, but couldn't think of any way to do it.

Thanks. Some serious help from @ovs to lose 30 bytes at the end tho so credit and respect due.

Ruby, 61 bytes
p Dir.glob("**/*.sh").sum{|x|open(x).grep(/^\s*[^#\s]/).size}

Try it online!

Röda + find, 98 bytes
{bufferedExec"find",".","-name","*.sh"|{|x|try readLines x}_|{|x|x~=`\s|#.*`,"";[1]if[#x>0]}_|sum}

Try it online!
Pure Röda, 129 bytes
{["."]|[_]if isFile(_1)else unpull(x)for x in[ls(_1)]|{|x|try readLines x if[x=~`.*\.sh`]}_|{|l|l~=`\s|#.*`,"";[1]if[#l>0]}_|sum}

Try it online!
Explanation:
{
["."]|                     /* Push "." to the stream */
                           /* For each _1 in the stream: */
[_]if isFile(_1)           /*   If _1 is a file, push it to the output stream */
else                       /*   Else (if _1 is a directory): */
unpull(x)for x in[ls(_1)]| /*     Push each file/dir in _1 to the *input stream* */
{|x|                       /* For each x in the stream: */
  try readLines x          /*   Push lines of x to the stream ignoring errors */
    if[x=~`.*\.sh`]        /*     if x ends in .sh */
}_|
{|l|                       /* For each l in the stream: */
  l~=`\s|#.*`,"";          /*   Remove whitespace and comments from l */
  [1]if[#l>0]              /*   Push 1 to the stream if l is not empty */
}_|
sum                        /* Sum all numbers in the stream */
}

C (clang), 248 209 bytes
#import <regex.h>
*l,i,j;*k;regex_t r;f(n,s,t){if(fnmatch("*.sh",n,0))return 0;k=fopen(n,"r");for(regcomp(&r,"^\\s*(#|$)",1);i=getline(&l,&i,k)>0;)regexec(&r,l,0,0,0)&&j++;}main(){ftw(".",f,1);printf("%d",j);}

Try it online!

Save one byte char i,j,*l; ==> char*l,i,j;

189 Bytes here

Why are you not counting your definition and include bytes?

@JonathanFrech, because it's just an argument, it could be any directory

I do not think that is legal I/O; some inclusions you perform are necessary and not counted.

See this default I/O consensus.

Suggest #import<regex.h> instead of #include <regex.h>. Also, can ditch ftw.h and fnmatch.h

PHP, 105 140 174 186 184 172 bytes
function f($d){foreach(glob("$d/*")as$n)$n[strlen($d)+1]^_^A&&$s+=!is_dir($n)?fnmatch("*.sh",$n)?preg_match_all("/^\s*[^#]/m",join(file($n))):0:f($n);return$s;}echo f(".");

skips hidden directories and files (name starts with a dot). Run with -nr.
breakdown
function f($d)
{
    # loop through directory entries
    foreach(glob("$d/*")as$n)
        # if name does not start with a dot
        $n[strlen($d)+1]^_^A
        # then increase sum:
        &&$s+=
            # not a directory?
            !is_dir($n)
                # filename matches pattern?
                ?fnmatch("*.sh",$n)
                    # count lines that are neither empty nor begin with a "#"
                    ?preg_match_all("/^\s*[^#]/m",join(file($n)))
                    # else 0
                    :0
                # is a directory: recurse
                :f($n)
        ;
    return$s;
}
echo f(".");

I could save two more bytes with fnmatch()*preg_match_all() instead of fnmatch()?preg_match_all():0;
and then another one with is_dir()?B:A instead of !is_dir()?A:B;
but that could make it insanely slow.

substr($n,-3)=='.sh' would be shorter than preg_match("/\.sh$/",$n).

Sorry, actually fnmatch('*.sh',$n) is even shorter.

@manatwork Wow I didn´t even know that function!

Yepp, weird thing. I only found out about it after used the similar File.fnmatch in Ruby for years.

Zsh, 57 bytes
for f (**/*.sh(D))a+=(${${(f)"$(<$f)"//[ 	]}%%#*})
<<<$#a

Try it online!
Almost beat bash+coreutils with native zsh constructs.
for f (**/*.sh(D))                    # the (D) flag enables globdots
   a+=(${${(f)"$(<$f)"//[   ]}%%#*})
 #       ${(f)"$(<$f)"       }        # read in file, split on newlines
 #       ${           //[   ]}        # remove all spaces and tabs
 #     ${                     %%#*}   # remove longest trailing #comment
 # a+=(                            )  # if append non-empty lines to array
<<<$#a                                # print array length

Zsh -oextendedglob -oglobstarshort -oglobdots, 51 bytes
for f (**.sh)a+=(${${(f)"$(<$f)"}/[ 	]#\#*})
<<<$#a

Try it online!
globstarshort allows **.sh, globdots removes the (D) flag, and extendedglob makes [ \t]# work like the regex [ \t]*.
Sidenote: I think some answers here don't match hidden files. If they can be safely ignored, then -oglobdots or (D) is not needed.

Nim, 114 bytes
import os,re
var c=0
for f in walkDirRec".":
 if f=~re".*\.sh$":
  for l in f.lines:c+=int l=~re"\s*[^#\s]"
echo c