or how to find invalid human batteries so they can be flushed.

In this post, part of the Bending .NET series, I look at compiling a complete list of valid and invalid C# identifier separators by using Roslyn. That is, by being lazy and instead of checking unicode specification just try every single character.

Source: wikimedia

Encoding information in file or directory names is a simple way of keeping details about a given file. For example, an image or an ONNX model and the conditions or origins of it. That is, for an ONNX model it might be something simple like the version of the ground truth that it was trained on. Assumming you, of course, version your ground truth. Defining a simple schema for this encoding using simple separators e.g. _, = can make it easy to then parse in a position independent manner like M=Surface_V=1.2.1.onnx. Now these files might allow embedding this information in them in some way, but sometimes it is just easier for reference to have it directly in the file name, for example.

Sometimes it is also nice to be able to embed these assets into .NET assemblies and have them exposed as simple properties. Preferably properties that have the same name as the file. However, C# naturally has more strict rules for identifier names than file names, which brings up the question what chars are valid separators in a C# identifier name?

The above link says:

• Identifiers must start with a letter or underscore (_).
• Identifiers may contain Unicode letter characters, decimal digit characters, Unicode connecting characters, Unicode combining characters, or Unicode formatting characters. For more information on Unicode categories, see the Unicode Category Database. You can declare identifiers that match C# keywords by using the @ prefix on the identifier. The @ is not part of the identifier name. For example, @if declares an identifier named if. These verbatim identifiers are primarily for interoperability with identifiers declared in other languages.

Now this is a perfectly fine answer, but I was just too lazy to go scouring through the Unicode Category Database 🦥. What I wanted was a single table of valid identifier separators. I googled for answers but came up short within a reasonable time, so instead I came up with this Program.cs (after some iterations as discussed below):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
using System.Diagnostics;
using System.Runtime.CompilerServices;
using System.Text;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;

var findByCompile = false;
var encoding = Encoding.UTF8;
Console.OutputEncoding = encoding;
Action<string> log = t => { Console.WriteLine(t); Trace.WriteLine(t); };
// Cache metadata reference since this reduces time significantly
var metadataReferences = new[] { MetadataReference.CreateFromFile(
    typeof(object).Assembly.Location) };

var validSeparators = new List<char>();
var invalidSeparators = new List<char>();

var validFileNameChars = new List<char>();
var invalidFileNameChars = Path.GetInvalidFileNameChars();
Array.Sort(invalidFileNameChars);

var stopwatch = Stopwatch.StartNew();
for (int i = char.MinValue; i <= char.MaxValue;)
{
    var c = (char)i;

    (IsValidSeparator(c) ? validSeparators : invalidSeparators).Add(c);

    if (Array.BinarySearch(invalidFileNameChars, c) < 0)
    { validFileNameChars.Add(c); }

    if (++i % 4096 == 0 && findByCompile) { log($"Compiled {i:D5} programs"); }
}
var elapsed_ms = stopwatch.ElapsedMilliseconds;

Write(validSeparators);
Write(invalidSeparators);
Write(validFileNameChars);
Write(invalidFileNameChars);

var totalCount = validSeparators.Count + invalidSeparators.Count;
log($"Found {validSeparators.Count} valid identifier separator chars.");
log($"Found {invalidSeparators.Count} invalid identifier separator chars.");
log($"Found {validFileNameChars.Count} valid file name chars.");
log($"Checked {totalCount} chars in {elapsed_ms} ms or " +
    $"{elapsed_ms / (double)totalCount:F3} ms per char.");

bool IsValidSeparator(char c) => findByCompile ? DoesCompile(c)
    : SyntaxFacts.IsValidIdentifier(Identifier(c));

bool DoesCompile(char c)
{
    var program = $"var {Identifier(c)} = 42;";
    var syntaxTree = CSharpSyntaxTree.ParseText(program);
    var compilation = CSharpCompilation.Create("assemblyName",
        new[] { syntaxTree }, metadataReferences,
        new CSharpCompilationOptions(OutputKind.ConsoleApplication));
    using var dllStream = new MemoryStream();
    var emitResult = compilation.Emit(dllStream);
    return emitResult.Success;
}

static string Identifier(char c) => $"_{c}_";

void Write(IReadOnlyList<char> chars,
    [CallerArgumentExpression("chars")] string fileName = "")
{
    const string baseDir = "../../../../";
    File.WriteAllText(baseDir + $"{fileName}.csv", ToCsv(chars), encoding);
    File.WriteAllText(baseDir + $"{fileName}.txt", ToTable(chars), encoding);
}

static string ToCsv(IReadOnlyList<char> chars) => string.Join(Environment.NewLine,
    new[] { CsvHeader() }.Concat(chars.Select(c => CsvLine(c))));
static string CsvHeader() => "Decimal,Hex,Identifier";
static string CsvLine(char c) => $"{(int)c:D5},0x{(int)c:X4},{Identifier(c)}";

static string ToTable(IReadOnlyList<char> chars) => string.Join(Environment.NewLine,
    new[] { TableHeader() }.Concat(chars.Select(c => TableLine(c))));
static string TableHeader() => $"|Decimal|Hex|Identifier|{Environment.NewLine}|-:|-:|-|";
static string TableLine(char c) => $"|{(int)c:D5}|`0x{(int)c:X4}`|`{Identifier(c)}`|";

For completeness here also the accompanying ValidIdentifierSeparators.csproj:

1
2
3
4
5
6
7
8
9
10
11
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.3.0" />
  </ItemGroup>
</Project>

As can be seen this references Microsoft.CodeAnalysis.CSharp or Roslyn as a nuget package.

The program is fairly self explanatory. As the astute reader might have observed, though, and as it happens with any software development you might start out going down the “wrong” rabbit hole. I certainly did that by first simply defining a full top-level statement single line program like var _{c}_ = 42; where c would be a given character and then use Roslyn to compile that program and check if this would succeed using CSharpCompilation.Emit. Carefully encapsulating the char in underscores _ to handle both the fact the first character is limited and if a char being tested is considered white space, which the final underscore handles.

Hence, I ended up compiling 65.536 programs to check if a given identifier was valid or not. This took about 300 s or about 5 minutes. Faster than scouring through the unicode database I am sure 😅 This seemed a bit slow… a quick profiling session revealed that:

1
MetadataReference.CreateFromFile(typeof(object).Assembly.Location)

took up most of the time. This basically parses and builds a “metadata reference” to the assembly containing object, which is the only assembly needed for the var _{c}_ = 42; program. This was the reason I moved this to the top of the program so it only happened once. This brought the run time down to ~22 s or 14-15x faster:

1
2
3
4
Found 50683 valid identifier separator chars.
Found 14853 invalid identifier separator chars.
Found 65495 valid file name chars.
Checked 65536 chars in 22076 ms or 0.337 ms per char.

Later, I finally found out Roslyn has the very nice SyntaxFacts and the obvious method to use namely IsValidIdentifer. Using this cut down run time to 5 ms or 60,000x faster than the initial program. 😆

1
2
3
4
Found 50683 valid identifier separator chars.
Found 14853 invalid identifier separator chars.
Found 65495 valid file name chars.
Checked 65536 chars in 5 ms or 0.000 ms per char.

I’ve kept both approaches in the above program and this also includes finding valid file name char’s for completeness. Without all this (and in particular all the output related code) you could have simply written the quick program below to trace valid separator chars. 🙄

1
2
3
4
5
6
7
8
9
for (int i = char.MinValue; i <= char.MaxValue; i++)
{
    var c = (char)i;
    var identifier = $"_{c}_";
    if (SyntaxFacts.IsValidIdentifier(identifier))
    {
        Trace.WriteLine(c);
    }
}

Compiled lists of valid and invalid separator chars and file name chars can be found in the table below. Note Jekyll (used for this blog) can’t handle the large markdown files so files have been given the extra .txt extension.

For reference in the next sections are the first valid separators until decimal 511 and first invalid separators. It is noteworthy - and of course natural - that most “ascii” non-digit/non-alpha characters are invalid as identifier separators. _ being the exception. I selected a few that might be used below, but these also have issues with being easily confused with the “normal” ascii range versions. Of course, this assume you use English as a language, so this is probably abusing other languages letters. They is another issue though…

… and that is some characters display as a square or rectangular box, or as a box with a dot, question mark or “x” inside depending on system or browser used. As can be seen below with screnshots from a desktop browser and on the phone. On the phone not all characters are displayed. I am no expert on the matter but as far as I know this is a font issue. So the end result is it is very hard to find any other good separator char than underscore _ in C# identifiers. However, you could then use multiples of this for separation instead e.g. M_Surface__V_1.2.1.onnx.

Desktop	Phone

Some Valid Identifier Separators

Some Invalid Identifier Separators

Related Stack Overflow Questions

• What characters are allowed in C# class name? this is what initially led me to go down the wrong rabbit hole, since among others the top answer is given below and I definitely didn’t want to go through those. 😅

  Essentially, any unicode character (including unicode escapes) in the character classes Lu, Ll, Lt, Lm, Lo, Nl, Mn, Mc, Nd, Pc, and Cf. The first character is an exception and it must be a letter (classes Lu, Ll, Lt, Lm, or Lo) or an underscore. Also, if the identifier is a keyword, you must stick an @ in front of it. The @ is optional otherwise.

• Is there a method in C# to check if a string is a valid identifier [duplicate] didn’t find this at first and although marked as duplicate the second answer is exactly what I was looking for:

  1
  2
  3
  var isValid = Microsoft.CodeAnalysis.CSharp.
      SyntaxFacts.IsValidIdentifier("I'mNotValid");
  Console.WriteLine(isValid);     // False
  

• How to determine if a string is a valid variable name? this is the question/answer the above was considered a duplicate for, but it does not mention SyntaxFacts.