Above png is very high resolution, but you can also download a pdf.

Sources

• .NET and C# Versions - 20th Anniversary ♥
• .NET and C# Versions - 7/11 Update
• .NET and C# Versions - 8/12 Update
• .NET and C# Versions - 9/13 Update
• What’s new in .NET 10
• What’s new in the .NET 10 runtime
• What’s new in .NET libraries for .NET 10
• What’s new in the SDK and tooling for .NET 10
• What’s new in C# 14
• Breaking Changes in .NET 10

Changes

• Add .NET 10.0 and C# 14.0

Both of these are available as free runners on GitHub and it is those virtual instances that are used for benchmarking Sep on ARM64, as I have no ARM64 hardware myself. It was the new availability of Cobalt 100 based GitHub runners that triggered me adding a new ARM NEON SIMD parser to Sep, as early benchmarks of this looked a bit “slow” compared to other CPUs.

Previously, Sep on ARM would use a cross-platform SIMD parser based on Vector128, which has been the case since early Sep e.g. Sep 0.2.0. Now Sep hits 9.5 GB/s on Apple M1 up from ~7 GB/s with the cross-platform SIMD parser, and ~6 GB/s on Cobalt 100 up from ~4 GB/s.

As seen last in Sep 0.10.0 - 21 GB/s CSV Parsing Using SIMD on AMD 9950X 🚀 this is not too bad compared to the insane performance of a large Zen 5 desktop core. I wonder what the performance would be on an Apple M4, but I do not have access to one. And it is pretty nice to be able to run such benchmarks as GitHub actions.

See v0.11.0 release for all changes for the release, and Sep README on GitHub for full details including detailed benchmarks.

Below I will dive into the new NEON SIMD parser, showing interesting code and assembly along the way. First I’ll show how low-level parsing performance of Sep has improved on Apple M1 due to this and how it compares to CsvHelper and Sylvan.Data.Csv.

Sep Performance on Apple M1

The above graph shows how Sep 0.11.0 improves the already stellar performance on Apple M1 compared to Sylvan and CsvHelper. Sep is now ~14x faster than CsvHelper at 9.5 GB/s vs 0.7 GB/s and 6.4x faster than Sylvan.

These numbers are, as seen before, for the package assets CSV data and the low level parse Rows only scope, see Sep README on GitHub or code on GitHub for details on this, but to recap for new readers:

• 1 char = 2 bytes
• Based on StringReader and string in memory to rule out any IO variation
• Data intentionally small and typically fits in L3 cache to be able to measure impact of even small code changes
• Single-threaded
• No quotes, no trimming, no unescaping, but still real-world CSV
• Just CSV parsing only, rows only
• Warm runs only as per usual with BenchmarkDotNet

In addition, since these benchmarks run on a virtual machine in the form of actions on a GitHub runner, there is higher variance. Usually locally I’d say there is about 5% variance but for GitHub runners this is about 10%, sometimes even much larger.

Hence, before Sep 0.11.0 using the cross-platform SIMD based parser on Apple M1 I’ve observed performance ranging from ~7000 - 7600 MB/s. With 0.11.0 and the NEON SIMD based parser this increases to ~8500-9700 MB/s, so the new parser is about 1.3-1.4x faster on Apple M1.

Sep Performance on Cobalt 100

The performance on Cobalt 100 before Sep 0.11.0 was only ~4 GB/s, which is what triggered me looking into improving ARM performance, since this seemed rather slow. After Sep 0.11.0 this has improved to ~6 GB/s, which is a nice 1.5x improvement. Still fairly slow compared to “performance” cores, but these ARM cores are not really designed for maximum single-threaded performance, but rather for high efficiency and density.

As can be also seen compared to Sylvan and CsvHelper, Sep is now ~11x faster than CsvHelper at 6.1 GB/s vs 0.6 GB/s and 4.2x faster than Sylvan on ARM Neoverse N2.

Disassembling via dotnet publish and dumpbin on Windows

Not having any ARM64 PC myself also meant adding a specific parser for this presented new challenges, as I could neither test/debug the code directly nor inspect the assembly for it directly using Disasmo as usual (maybe that is possible but I do not know how). Debugging wasn’t that much of an issue, since I only had to do minor changes to SIMD code in 0.11.0 and few issues arose, but inspecting the assembly was more important to be sure the generated machine code was in line with expected instructions.

Now I could have tried using a GitHub action on e.g. Apple M1 runner to get to the assembly, but that would have been a terrible dev loop and I am too impatient for that.

Instead, I figured out I could use NativeAOT via dotnet publish of a small test executable and have this generate an executable for win-arm64, which is possible on Windows if you have the right development tools installed e.g. “Visual Studio 2022, including the Desktop development with C++ workload with all default components.” And then I could use dumpbin to disassemble the resulting executable as shown in the script below.

1
2
3
4
5
6
7
8
9
param(
    [string]$runtime = "win-arm64"
)
dotnet publish src/Sep.Tester/Sep.Tester.csproj `
  -c Release -r "$runtime" -f net9.0 --self-contained true `
  /p:PublishAot=true /p:DebugSymbols=true
dumpbin /DISASM /SYMBOLS `
  "artifacts\publish\Sep.Tester\release_net9.0_$runtime\Sep.Tester.exe" >` 
  "artifacts\publish\Sep.Tester\release_net9.0_$runtime\disassembly.asm"

The disassembly.asm file is then a text file that I can keep open in VS or WinMerge and in which I manually search for relevant class names and methods, which are available since there are debug symbols, to find the assembly I am interested in. This also allows me to compare x64 vs ARM64 if need be.

Note that there may be differences in the assembly generated with NativeAOT vs JIT in general, but for the specific places where I am interested in this for Sep there hasn’t been any.

ARM SIMD - Cross-platform vs AdvSimd

Let’s take a look at the cross-platform SIMD C# code previously used on ARM (and any other non-x86 platform with Vector128 hardware acceleration). Basic approach has been discussed before also in Sep 0.10.0 blog post and I am showing only partial inner loop code. For full code go to nietras/Sep on GitHub.

Since cross-platform Vector128 SIMD does not (currently) have support for saturated conversion/narrowing of 16-bit unsigned integers (e.g. char) to 8-bit unsigned integers (this is coming in .NET 10 I believe with NarrowWithSaturation methods), this is done by using Min before Narrow as can be seen in the C# code below. Additionally, this uses ExtractMostSignificantBits for MoveMask.

SepParserVector128NrwCmpExtMsbTzcnt

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
ref var charsRef = ref Add(ref charsOriginRef, (uint)charsIndex);
ref var byteRef = ref As<char, byte>(ref charsRef);
var v0 = ReadUnaligned<VecUI16>(ref byteRef);
var v1 = ReadUnaligned<VecUI16>(ref Add(ref byteRef, VecUI8.Count));
var limit0 = Vec.Min(v0, max);
var limit1 = Vec.Min(v1, max);
var bytes = Vec.Narrow(limit0, limit1);

var nlsEq = Vec.Equals(bytes, nls);
var crsEq = Vec.Equals(bytes, crs);
var qtsEq = Vec.Equals(bytes, qts);
var spsEq = Vec.Equals(bytes, sps);

var lineEndings = nlsEq | crsEq;
var lineEndingsSeparators = spsEq | lineEndings;
var specialChars = lineEndingsSeparators | qtsEq;

// Optimize for the case of no special character
var specialCharMask = MoveMask(specialChars);
if (specialCharMask != 0u)

The assembly for this, based on using NativeAOT and dumpbin as discussed above for ARM64 with NEON SIMD support, is shown below. What is interesting here is both the instructions needed for narrowing but also the many instructions needed for the MoveMask operation, as ARM NEON does not have an equivalent to this built-in. A clear oversight by ARM…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
AD405935  ldp         q21,q22,[x9]
6E706EB5  umin        v21.8h,v21.8h,v16.8h
0E212AB5  xtn         v21.8b,v21.8h
6E706ED6  umin        v22.8h,v22.8h,v16.8h
4E212AD5  xtn2        v21.16b,v22.8h
6E318EB6  cmeq        v22.16b,v21.16b,v17.16b
6E328EB7  cmeq        v23.16b,v21.16b,v18.16b
6E338EB8  cmeq        v24.16b,v21.16b,v19.16b
6E348EB5  cmeq        v21.16b,v21.16b,v20.16b
4EB71ED6  orr         v22.16b,v22.16b,v23.16b
4EB61EB6  orr         v22.16b,v21.16b,v22.16b
4EB81ED7  orr         v23.16b,v22.16b,v24.16b
4F04E418  movi        v24.16b,#0x80
4E381EF7  and         v23.16b,v23.16b,v24.16b
9C0019F9  ldr         q25,000000014007B140
6E3946F7  ushl        v23.16b,v23.16b,v25.16b
4EB71EFA  mov         v26.16b,v23.16b
0E31BB5A  addv        b26,v26.8b
0E013F4D  umov        w13,v26.b[0]
6E1742F7  ext8        v23.16b,v23.16b,v23.16b,#8
0E31BAF7  addv        b23,v23.8b
0E013EEE  umov        w14,v23.b[0]
2A0E21AD  orr         w13,w13,w14,lsl #8
2A0D03ED  mov         w13,w13
B4FFFC4D  cbz         x13,000000014007ADB4

Many people have noticed this and in Links below I have listed a few resources that discuss this. One of which is Fitting My Head Through The ARM Holes or: Two Sequences to Substitute for the Missing PMOVMSKB Instruction on ARM NEON by Geoff Langdale, a notable figure in the SIMD community and one of the authors on the sadly incomplete simdcsv. The blog post discusses a way to do “move mask” efficiently in bulk for four 128-bit registers in one go. Please read that blog post for more, since I won’t cover the trickery here 😊.

This is the approach I have added in SepParserAdvSimdNrwCmpOrBulkMoveMaskTzcnt, with key code shown below, in addition to using efficient saturated conversion from 16-bit to 8-bit, which is supported in ARM NEON. However, to do that here we have to load a total of 8 x Vector128s in each loop first, and then narrow/convert to 4 x Vector128, before doing comparisons, or’ing and finally extract bits to get a mask. This means this parser is handling 1024 bits or 64 x chars at a time, which is the same as the AVX-512 based parser. And the logic is based on comparisons resulting in all bits being 1 for each byte where the comparison is equal.

SepParserAdvSimdNrwCmpOrBulkMoveMaskTzcnt

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
ref var charsRef = ref Add(ref charsOriginRef, (uint)charsIndex);
var bytes0 = ReadNarrow(ref charsRef);
var bytes1 = ReadNarrow(ref Add(ref charsRef, VecUI8.Count * 1));
var bytes2 = ReadNarrow(ref Add(ref charsRef, VecUI8.Count * 2));
var bytes3 = ReadNarrow(ref Add(ref charsRef, VecUI8.Count * 3));

var nlsEq0 = AdvSimd.CompareEqual(bytes0, nls);
var crsEq0 = AdvSimd.CompareEqual(bytes0, crs);
var qtsEq0 = AdvSimd.CompareEqual(bytes0, qts);
var spsEq0 = AdvSimd.CompareEqual(bytes0, sps);
var lineEndings0 = AdvSimd.Or(nlsEq0, crsEq0);
var lineEndingsSeparators0 = AdvSimd.Or(spsEq0, lineEndings0);
var specialChars0 = AdvSimd.Or(lineEndingsSeparators0, qtsEq0);

var nlsEq1 = AdvSimd.CompareEqual(bytes1, nls);
var crsEq1 = AdvSimd.CompareEqual(bytes1, crs);
var qtsEq1 = AdvSimd.CompareEqual(bytes1, qts);
var spsEq1 = AdvSimd.CompareEqual(bytes1, sps);
var lineEndings1 = AdvSimd.Or(nlsEq1, crsEq1);
var lineEndingsSeparators1 = AdvSimd.Or(spsEq1, lineEndings1);
var specialChars1 = AdvSimd.Or(lineEndingsSeparators1, qtsEq1);

var nlsEq2 = AdvSimd.CompareEqual(bytes2, nls);
var crsEq2 = AdvSimd.CompareEqual(bytes2, crs);
var qtsEq2 = AdvSimd.CompareEqual(bytes2, qts);
var spsEq2 = AdvSimd.CompareEqual(bytes2, sps);
var lineEndings2 = AdvSimd.Or(nlsEq2, crsEq2);
var lineEndingsSeparators2 = AdvSimd.Or(spsEq2, lineEndings2);
var specialChars2 = AdvSimd.Or(lineEndingsSeparators2, qtsEq2);

var nlsEq3 = AdvSimd.CompareEqual(bytes3, nls);
var crsEq3 = AdvSimd.CompareEqual(bytes3, crs);
var qtsEq3 = AdvSimd.CompareEqual(bytes3, qts);
var spsEq3 = AdvSimd.CompareEqual(bytes3, sps);
var lineEndings3 = AdvSimd.Or(nlsEq3, crsEq3);
var lineEndingsSeparators3 = AdvSimd.Or(spsEq3, lineEndings3);
var specialChars3 = AdvSimd.Or(lineEndingsSeparators3, qtsEq3);

// Optimize for the case of no special character
var specialCharMask = MoveMask(specialChars0, specialChars1,
                               specialChars2, specialChars3);
if (specialCharMask != 0u)

1
2
3
4
5
6
7
8
9
10
11
[MethodImpl(MethodImplOptions.AggressiveInlining)]
static VecUI8 ReadNarrow(ref char charsRef)
{
    ref var byteRef = ref As<char, byte>(ref charsRef);
    var v0 = ReadUnaligned<VecUI16>(ref byteRef);
    var v1 = ReadUnaligned<VecUI16>(ref Add(ref byteRef, VecUI8.Count));

    var r0 = AdvSimd.ExtractNarrowingSaturateLower(v0);
    var r1 = AdvSimd.ExtractNarrowingSaturateUpper(r0, v1);
    return r1;
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[MethodImpl(MethodImplOptions.AggressiveInlining)]
internal static nuint MoveMask(VecUI8 p0, VecUI8 p1, VecUI8 p2, VecUI8 p3)
{
    var bitmask = Vec.Create(
        0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80,
        0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80
    );

    var t0 = AdvSimd.And(p0, bitmask);
    var t1 = AdvSimd.And(p1, bitmask);
    var t2 = AdvSimd.And(p2, bitmask);
    var t3 = AdvSimd.And(p3, bitmask);

    var sum0 = AdvSimd.Arm64.AddPairwise(t0, t1);
    var sum1 = AdvSimd.Arm64.AddPairwise(t2, t3);
    sum0 = AdvSimd.Arm64.AddPairwise(sum0, sum1);
    sum0 = AdvSimd.Arm64.AddPairwise(sum0, sum0);

    return (nuint)sum0.AsUInt64().GetElement(0);
}

The assembly for this can be seen below. Given this now processes a lot more in one go there are a lot more repeated instructions, but then we also reduce the relative number of instructions per bit in the mask generated since we handle more at a time. The performance benefits are clear given the numbers shown above. All thanks to Geoff Langdale 🙏

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
AD405534  ldp         q20,q21,[x9]
2E214A94  uqxtn       v20.8b,v20.8h
6E214AB4  uqxtn2      v20.16b,v21.8h
9100812D  add         x13,x9,#0x20
AD4059B5  ldp         q21,q22,[x13]
2E214AB5  uqxtn       v21.8b,v21.8h
6E214AD5  uqxtn2      v21.16b,v22.8h
9101012D  add         x13,x9,#0x40
AD405DB6  ldp         q22,q23,[x13]
2E214AD6  uqxtn       v22.8b,v22.8h
6E214AF6  uqxtn2      v22.16b,v23.8h
9101812D  add         x13,x9,#0x60
AD4061B7  ldp         q23,q24,[x13]
2E214AF7  uqxtn       v23.8b,v23.8h
6E214B17  uqxtn2      v23.16b,v24.8h
6E308E98  cmeq        v24.16b,v20.16b,v16.16b
6E318E99  cmeq        v25.16b,v20.16b,v17.16b
6E328E9A  cmeq        v26.16b,v20.16b,v18.16b
6E338E94  cmeq        v20.16b,v20.16b,v19.16b
4EB91F18  orr         v24.16b,v24.16b,v25.16b
4EB81E98  orr         v24.16b,v20.16b,v24.16b
4EBA1F19  orr         v25.16b,v24.16b,v26.16b
6E308EBA  cmeq        v26.16b,v21.16b,v16.16b
6E318EBB  cmeq        v27.16b,v21.16b,v17.16b
6E328EBC  cmeq        v28.16b,v21.16b,v18.16b
6E338EB5  cmeq        v21.16b,v21.16b,v19.16b
4EBB1F5A  orr         v26.16b,v26.16b,v27.16b
4EBA1EBA  orr         v26.16b,v21.16b,v26.16b
4EBC1F5B  orr         v27.16b,v26.16b,v28.16b
6E308EDC  cmeq        v28.16b,v22.16b,v16.16b
6E318EDD  cmeq        v29.16b,v22.16b,v17.16b
6E328EDE  cmeq        v30.16b,v22.16b,v18.16b
6E338ED6  cmeq        v22.16b,v22.16b,v19.16b
4EBD1F9C  orr         v28.16b,v28.16b,v29.16b
4EBC1EDC  orr         v28.16b,v22.16b,v28.16b
4EBE1F9D  orr         v29.16b,v28.16b,v30.16b
6E308EFE  cmeq        v30.16b,v23.16b,v16.16b
6E318EFF  cmeq        v31.16b,v23.16b,v17.16b
6E328EE7  cmeq        v7.16b,v23.16b,v18.16b
6E338EF7  cmeq        v23.16b,v23.16b,v19.16b
4EBF1FDE  orr         v30.16b,v30.16b,v31.16b
4EBE1EFE  orr         v30.16b,v23.16b,v30.16b
4EA71FDF  orr         v31.16b,v30.16b,v7.16b
9C0019E7  ldr         q7,000000014007AD30
4E271F39  and         v25.16b,v25.16b,v7.16b
4E271F7B  and         v27.16b,v27.16b,v7.16b
4E3BBF39  addp        v25.16b,v25.16b,v27.16b
4E271FBB  and         v27.16b,v29.16b,v7.16b
4E271FFD  and         v29.16b,v31.16b,v7.16b
4E3DBF7B  addp        v27.16b,v27.16b,v29.16b
4E3BBF39  addp        v25.16b,v25.16b,v27.16b
4E39BF39  addp        v25.16b,v25.16b,v25.16b
4E083F2D  mov         x13,v25.d[0]
B4FFF8AD  cbz         x13,000000014007A930

Links

Interesting links related to ARM SIMD. The first one explains the approach adopted by Sep for efficient bulk move mask in ARM NEON. Others, can be used for inspiration or perhaps future improvements to Sep.

• Fitting My Head Through The ARM Holes or: Two Sequences to Substitute for the Missing PMOVMSKB Instruction on ARM NEON by Geoff Langdale.
• Pruning Spaces Faster on ARM Processors with Vector Table Lookups by Daniel Lemire.
• Bit twiddling with Arm Neon: beating SSE movemasks, counting bits and more by Danila Kutenin.
• ARM Intrinsics

That’s all!

First, a bit of color on why we need this for our machine learning workflow.

Use Case and Problem

At work we train all our machine learning image models in house on custom build servers featuring GPUs like NVIDIA RTX 4090. We have a lot of data - collected from production sites all over the world - and this data is usually split in two.

1. Ground truth annotations are typically defined in csv-files that are stored in git repositories and published as versioned NuGet packages. These packages are then consumed by different pipelines that are run as Azure Pipelines on the servers. We then use Renovate to automatically bump versions of these.
2. Images are stored on a file server on a network share, and typically also cached at specific resolutions locally on the servers to speed up training.

At the same time the path to and file name of the images usually follows a schema so the path alone contains information relevant to a given image. This means paths can get quite long. Certainly above 260 characters on the file server.

Example path schema for directory:

1
\\fileserver\Pipelines\<PROJECTNAME>\<SETDATETIME>_<SETNAME>_<SITE>\<STATION>\

Example file name schema:

1
<DATETIME>_Camera=<CAMERANAME>_Id=<ID>_<DETAILS>.png

For example:

1
20250102.123456.789_Camera=Primary_Id=9999999_x=0123_y=0234_w=1234_h=2345.png

This is a simple pragmatic setup that works very well and a setup we have iterated on over many years and runs very smoothly. Each pipeline is a separate git repo that is easy to get started with by simply cloning and hitting F5 in Visual Studio or similar for local running.

As part of the output of the pipelines we generate a lot of data either as simple csv-files, plots png-files or interactive reports in the form of html-files with inline JavaScript and data.

Below is an example showing box plots including all samples as dots for a regression model with different “levels”. The box plots are interactive and when pointing on a singular dot (e.g. a “minimum” outlier) the image for that sample will be shown on the right. Usually, but not here since if the path is too long (> 260 chars) the browser won’t show it.

This is where there can be issues since browsers do not support showing long file paths, e.g. see chromium: Long file path handling not working on Windows. And browsers do not support extended length prefix \\?\ that one can otherwise use to escape the MAX_PATH 260 char limit in Win32 APIs.

Now if you SHIFT right click on a file in Windows Explorer and click Copy as path you will get the 8.3 short file name path if the file path is longer than MAX_PATH, as shown below both for a UNC file path and a local path name.

Copy as path for a local file resulting in C:\Temp\LONGFI~1\VERYLO~1\202501~1.PNG:

Copy as path for a network share file resulting in \\files\Pipelines\LXF59T~G\V1WO8N~7\290O13~C.PNG:

Similarly, if you right click and select Open with… and select a browser like Chrome then it will use the short path name for long file paths.

Hence, to fix the above interactive box plots we need to get the short path name in C#, as this is what we use to generate the html-files. This has the added benefit of reducing the amount of data we have to store in the html-file since the path name is shorter (we already do some custom path compression on this using a simple common prefix algorithm). Our data sets can have hundreds of thousands of images so every byte counts.

Incidentally, it would have been nice if the browsers would support loading gzip’ed html e.g. files directly, but as far as I know they do not, and spinning up a web server just for this just seemed overkill.

So let’s take a quick look at Windows 8.3 short paths and some example code on how get them in C# for both local and UNC paths.

What Are Windows 8.3 Short Paths?

Windows 8.3 short paths (also called “short file names” or SFN) are a legacy feature from MS-DOS and early Windows versions. They provide a way to represent long file and directory names (introduced with Windows 95 and NTFS) in a format compatible with older software that only supports 8-character filenames and 3-character extensions (e.g., MYDOCU~1.TXT). See Naming Files, Paths, and Namespaces - Short vs. Long Names or wikipedia 8.3 filename for more. Example:

• Long name: C:\Program Files\My Application\readme.txt
• Short name: C:\PROGRA~1\MYAPPL~1\README.TXT

How Are 8.3 Short Paths Stored in NTFS?

NTFS stores both the long and short (8.3) names for each file and directory, if 8.3 name generation is enabled. The short name is generated when a file is created, following specific rules to ensure uniqueness.

• Short names are stored as metadata in the NTFS file system.
• You can disable 8.3 name creation for performance reasons, but this may break compatibility with legacy applications.

Local Paths vs. UNC Paths

• Local paths refer to files on a local drive, e.g., C:\folder\file.txt.
• UNC (Universal Naming Convention) paths refer to files on a network share, e.g., \\server\share\folder\file.txt.

Key differences:

• Local paths are handled directly by the local file system.
• UNC Paths are resolved over the network, and some Windows APIs behave differently or have limitations with UNC paths.

Why Browsers and Some Tools Fail with Long Network Paths

Windows has a traditional MAX_PATH limit of 260 characters for file paths. While modern Windows versions and .NET can support longer paths (with configuration), many tools - including browsers - do not support long UNC paths due to:

• Lack of support for the \\?\ extended-length path prefix.
• Network shares (UNC) are not always handled with the same APIs as local paths.
• Browsers use standard Windows APIs that may not support long paths or UNC paths.

Getting the 8.3 Short Path Programmatically

For Local Paths

Use the Windows API function GetShortPathName:

1
2
[DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
static extern uint GetShortPathName(string lpszLongPath, StringBuilder lpszShortPath, uint cchBuffer);

• Works for local file system paths (e.g., C:\...).
• Does not work for UNC paths.

For UNC Paths (Network Shares)

Use FindFirstFileW to retrieve the 8.3 name for each segment of the path:

• Iterate over each directory/file segment in the UNC path.
• For each, call FindFirstFileW and use the cAlternateFileName field from the returned WIN32_FIND_DATA structure.
• Rebuild the path using the 8.3 names.

Code for this including struct definitions is shown below.

Summary Table

Win32ShortPath Class

The Win32ShortPath class shown below is factored into the following main methods.

• GetShortPath - Entry point. Normalizes the path, checks if it exists, and dispatches to the correct method:
  • For local paths, calls GetByGetShortPathName.
  • For UNC paths, calls GetByFindFirstFile.
• GetByGetShortPathName - Uses the GetShortPathName Win32 API to get the short path for local files and directories. GetShortPathName requires long paths to be “escaped” with the extended length prefix \\?\.
• GetByFindFirstFile - For UNC paths, splits the path into segments and uses FindFirstFileW to retrieve the 8.3 short name for each segment, rebuilding the full short path.
  • AppendPartsByFindFirstFile - Iterates through each path segment, using FindFirstFileW to get the short name (cAlternateFileName) if available.

Note that it does not handle all edge cases including using / as directory separator or similar. It is also not the version we use internally as here we already have the path defined as parts (typically) matching CSV-columns.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
using System.ComponentModel;
using System.Runtime.InteropServices;
using System.Text;

public static partial class Win32ShortPath
{
    const int MaxPathLength = 260;
    const int UncPrefixPartCount = 2;
    static readonly char DirectorySeparator = Path.DirectorySeparatorChar;
    const string UncPrefix = @"\\";
    const string ExtendedLengthPrefix = @"\\?\";

    /// <summary>
    /// Returns the short (8.3) path for any file or directory if 
    /// available, covering both local and UNC paths.
    /// </summary>
    public static string GetShortPath(string longPath)
    {
        if (string.IsNullOrWhiteSpace(longPath))
        { throw new ArgumentNullException(nameof(longPath)); }

        // Normalize
        longPath = Path.GetFullPath(longPath);

        if (!File.Exists(longPath) && !Directory.Exists(longPath))
        { throw new FileNotFoundException("Path not found", longPath); }

        var isUnc = longPath.StartsWith(UncPrefix, StringComparison.Ordinal);
        if (isUnc) { return GetByFindFirstFile(longPath); }

        var isLong = longPath.Length > MaxPathLength;
        if (isLong)
        {
            // Ensure starts with extended length prefix
            var extendedPrefix = longPath.StartsWith(ExtendedLengthPrefix,
                StringComparison.Ordinal);
            if (!extendedPrefix)
            {
                longPath = ExtendedLengthPrefix + longPath;
            }
            var shortPathName = GetByGetShortPathName(longPath);
            // Remove the extended length prefix if added
            if (!extendedPrefix)
            {
                shortPathName = shortPathName[ExtendedLengthPrefix.Length..];
            }
            return shortPathName;
        }
        else
        {
            return GetByGetShortPathName(longPath);
        }
    }

    // Local drive (C:\…), use GetShortPathName directly, for long prefix with \\?\
    static string GetByGetShortPathName(string longPath)
    {
        var sb = new StringBuilder(MaxPathLength);
        uint size = GetShortPathName(longPath, sb, (uint)sb.Capacity);
        if (size == 0) { throw new Win32Exception(Marshal.GetLastWin32Error()); }
        return sb.ToString();
    }

    // UNC path: \\server\share\…\file.ext, use FindFirstFile iteratively
    static string GetByFindFirstFile(string longPath)
    {
        // Rebuild segment by segment, use FindFirstFile for the 
        // 8.3 name of each.
        var parts = longPath.TrimStart(DirectorySeparator)
                            .Split(DirectorySeparator);
        if (parts.Length < UncPrefixPartCount)
        { throw new ArgumentException($"Invalid UNC path '{longPath}'", nameof(longPath)); }

        var sb = new StringBuilder(MaxPathLength);

        // Re‑prefix with \\server\share
        sb.Append(UncPrefix)
          .Append(parts[0])
          .Append(DirectorySeparator)
          .Append(parts[1]);

        AppendPartsByFindFirstFile(parts, UncPrefixPartCount, sb);

        return sb.ToString();
    }

    static void AppendPartsByFindFirstFile(string[] parts, int partStart,
                                           StringBuilder sb)
    {
        for (int i = partStart; i < parts.Length; i++)
        {
            var currentPart = parts[i];
            var pathToQuery = $"{sb}{DirectorySeparator}{currentPart}";
            var findHandle = FindFirstFileW(pathToQuery, out var findData);
            sb.Append(DirectorySeparator);
            if (findHandle != IntPtr.Zero)
            {
                FindClose(findHandle);
                // If there's an alternate (8.3) name, use it
                var alternatePart = findData.cAlternateFileName;
                currentPart = string.IsNullOrEmpty(alternatePart)
                    ? currentPart : alternatePart;
            }
            sb.Append(currentPart);
        }
    }

    [DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
    static extern uint GetShortPathName(string lpszLongPath,
        StringBuilder lpszShortPath, uint cchBuffer);

    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode, Pack = 1)]
    struct WIN32_FIND_DATA
    {
        const int AlternateLength = 14;

        public FileAttributes dwFileAttributes;
        public long ftCreationTime;
        public long ftLastAccessTime;
        public long ftLastWriteTime;
        public uint nFileSizeHigh;
        public uint nFileSizeLow;
        public uint dwReserved0;
        public uint dwReserved1;
        [MarshalAs(UnmanagedType.ByValTStr, SizeConst = MaxPathLength)]
        public string cFileName;
        [MarshalAs(UnmanagedType.ByValTStr, SizeConst = AlternateLength)]
        public string cAlternateFileName; // The 8.3 name
    }

    [DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
    static extern IntPtr FindFirstFileW(string lpFileName,
        out WIN32_FIND_DATA lpFindFileData);

    [DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
    static extern IntPtr FindFirstFileW(StringBuilder lpFileName,
        out WIN32_FIND_DATA lpFindFileData);

    [DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
    static extern bool FindClose(IntPtr hFindFile);
}

Example Program

Below is a simple example program using this with some (truncated) example paths that I created for testing:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
using System.Diagnostics;
Action<string> log = t => { Console.WriteLine(t); Trace.WriteLine(t); };

ReadOnlySpan<Test> tests =
[
    // Unc path long
    new(@"\\files\Pipelines\LongFilePathTest\VeryLong...\20250102.123456.789_Camera=Primary_Id=9999999_x=0123_y=0234_w=1234_h=2345.png",
        @"\\files\Pipelines\LXF59T~G\V1WO8N~7\290O13~C.PNG"),
    // Local path long
    new(@"C:\Temp\LongFilePathTest\VeryLong...\20250102.123456.789_Camera=Primary_Id=9999999_x=0123_y=0234_w=1234_h=2345.png",
        @"C:\Temp\LONGFI~1\VERYLO~1\202501~1.PNG"),
    // Local path long with extended prefix
    new(@"\\?\C:\Temp\LongFilePathTest\VeryLong...\20250102.123456.789_Camera=Primary_Id=9999999_x=0123_y=0234_w=1234_h=2345.png",
        @"\\?\C:\Temp\LONGFI~1\VERYLO~1\202501~1.PNG"),
    // Local path not long
    new(@"C:\Temp\LongFilePathTest\NotLong\20250102.123456.789_Camera=Primary_Id=9999999_x=0123_y=0234_w=1234_h=2345.png",
        @"C:\Temp\LONGFI~1\NotLong\202501~1.PNG"),
];
foreach (var t in tests)
{
    log($"Long Path: '{t.Path}'");
    log($"File Exists: {File.Exists(t.Path)}");
    var shortPath = Win32ShortPath.GetShortPath(t.Path);
    log($"Short Path: '{shortPath}'");
    log(t.ShortPathExpected == shortPath ? "PASSED" : "FAILED");
    log("");
}

public record Test(string Path, string ShortPathExpected);

When run, this will output the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Long Path: '\\files\Pipelines\LongFilePathTest\VeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVery\20250102.123456.789_Camera=Primary_Id=9999999_x=0123_y=0234_w=1234_h=2345.png'
File Exists: True
Short Path: '\\files\Pipelines\LXF59T~G\V1WO8N~7\290O13~C.PNG'
PASSED

Long Path: 'C:\Temp\LongFilePathTest\VeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVery\20250102.123456.789_Camera=Primary_Id=9999999_x=0123_y=0234_w=1234_h=2345.png'
File Exists: True
Short Path: 'C:\Temp\LONGFI~1\VERYLO~1\202501~1.PNG'
PASSED

Long Path: '\\?\C:\Temp\LongFilePathTest\VeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVeryLongVery\20250102.123456.789_Camera=Primary_Id=9999999_x=0123_y=0234_w=1234_h=2345.png'
File Exists: True
Short Path: '\\?\C:\Temp\LONGFI~1\VERYLO~1\202501~1.PNG'
PASSED

Long Path: 'C:\Temp\LongFilePathTest\NotLong\20250102.123456.789_Camera=Primary_Id=9999999_x=0123_y=0234_w=1234_h=2345.png'
File Exists: True
Short Path: 'C:\Temp\LONGFI~1\NotLong\202501~1.PNG'
PASSED

Considerations

8.3 short path names on Windows are not guaranteed to be stable across repeated file deletions and recreations — even if the long file name remains the same.

• 8.3 short names are dynamically assigned by the Windows file system (usually NTFS) and are not deterministic. If you delete a file and create a new one with the same long name, the 8.3 name might:
  • Remain the same (if no name conflict occurs),
  • Change (if the name was reused or reclaimed for another file),
  • Or not be created at all (if 8.3 name generation is disabled or restricted).
• Name conflicts are resolved by appending a numeric suffix (e.g., LONGFI~1.TXT, LONGFI~2.TXT), and the exact suffix depends on what already exists in the directory at the time of file creation.

Links and Further Reading

• Naming Files, Paths, and Namespaces (Microsoft Docs)
• Maximum Path Length Limitation (Microsoft Docs)
• GetShortPathName function (Microsoft Docs)
• FindFirstFileW function (Microsoft Docs)
• Long Filename Specification
• The Definitive Guide on Win32 to NT Path Conversion
• Why Disabling the Creation of 8.3 DOS File Names Will Not Improve Performance. Or Will It?

That’s all!

It handles skipping already downloaded packages and for our case can handle checking several feeds and ~10,000 packages and versions in less than one minute when no new packages found. No further details are provided here, but the script is available below, I thought perhaps someone might find it useful as a reference and the code is pretty straighforward.

Note that this only works for organization feeds not project feeds.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
trigger: none

schedules:
  # 01:00 UTC every Sunday
  - cron: "0 1 * * 0"
    displayName: Weekly NuGet Backup
    branches:
      include:
        - main
    always: true

pool:
  name: 'Default'

variables:
  ORGANIZATION_NAME: 'YOURORGNAME'
  NETWORK_PATH: '\\YOURLOCALPATH' # UNC path to your network share

steps:
  - task: PowerShell@2
    displayName: Download NuGet packages from all feeds
    env:
      SYSTEM_ACCESSTOKEN: $(System.AccessToken)
    inputs:
      targetType: inline
      script: |
        $ErrorActionPreference = 'Stop'

        $org = "$(ORGANIZATION_NAME)"
        $networkBase = "$(NETWORK_PATH)"
        $token = $env:SYSTEM_ACCESSTOKEN
        $headers = @{ Authorization = "Bearer $token" }

        # Get all feeds
        $feedsUrl = "https://feeds.dev.azure.com/$org/_apis/packaging/feeds?api-version=7.1-preview.1"
        Write-Host "Fetching all feeds from $feedsUrl"
        $feedsResp = Invoke-RestMethod -Uri $feedsUrl -Headers $headers
        $feeds = $feedsResp.value

        Write-Host "Feeds count $($feeds.Count)"
        if ($feeds.Count -eq 0) {
          Write-Host "No feeds found."
          exit 0
        }

        foreach ($feed in $feeds) {
          $feedName = $feed.name
          $feedId = $feed.id
          $feedPath = Join-Path $networkBase $feedName

          Write-Host "=================================================================="
          Write-Host "====== '$($feed.name)'"
          Write-Host "=================================================================="

          $fetchBaseUrl = "https://feeds.dev.azure.com/$org/_apis/packaging/feeds/$feedId/packages?protocolType=NuGet&api-version=7.1"
          $downloadBaseUrl = "https://pkgs.dev.azure.com/$org/_apis/packaging/feeds/$feedId/nuget/packages/"
          $pageSize = 50000
          $skip = 0
          $allPkgs = @()
          $page = 1

          # Fetch package names and versions from the feed
          Write-Host "Fetching packages from feed '$feedName'..."

          while ($true) {
            $url = "$fetchBaseUrl&includeAllVersions=true&`$top=$pageSize&`$skip=$skip"
            Write-Host "Requesting page $($page) from $($url)"
            $resp = Invoke-RestMethod -Uri $url -Headers $headers

            if ($resp.value.Count -eq 0) {
              break
            }

            $allPkgs += $resp.value
            Write-Host "Fetched $($resp.value.Count) packages in page $($page)."
            $skip += $resp.value.Count
            $page++
          }

          Write-Host "Package names found $($allPkgs.Count) in feed '$feedName'"

          if ($allPkgs.Count -eq 0) {
            Write-Host "WARNING No packages found in feed '$feedName'."
            continue
          }

          # Ensure the target directory exists
          if (-not (Test-Path $feedPath)) {
            Write-Host "Creating directory '$feedPath'..."
            New-Item -ItemType Directory -Path $feedPath -Force | Out-Null
          } else {
            Write-Host "Directory '$feedPath' already exists."
          }

          # Download packages (skip already exists)
          $totalPackagesCount = 0
          foreach ($pkg in $allPkgs) {
            Write-Host "====== '$($pkg.name)' versions found $($pkg.versions.Count) ======"
            foreach ($ver in $pkg.versions) {
              $nupkgFileName = "$($pkg.name).$($ver.version).nupkg"
              $nupkgPath = Join-Path $feedPath $nupkgFileName

              if (Test-Path $nupkgPath) {
                Write-Host "Skipping '$($nupkgPath)' (already exists)"
                continue
              }

              $downloadUrl = "$($downloadBaseUrl)$($pkg.name)/versions/$($ver.version)/content?api-version=7.1-preview"
              Write-Host "Downloading '$($nupkgFileName)' to '$feedPath' from '$($downloadUrl)'"
              Invoke-RestMethod -Uri $downloadUrl -Headers $headers -OutFile $nupkgPath
            }
            $totalPackagesCount += $pkg.versions.Count
          }
          Write-Host "Total packages found (already downloaded or downloaded) $($totalPackagesCount) in feed '$feedName'"
        }

That’s all!

See v0.10.0 release for all changes for the release, and Sep README on GitHub for full details.

In this blog post, I will dive into how .NET 9.0 machine code for AVX-512 is sub-optimal and what changes were made to speed up Sep for AVX-512 by circumventing this, showing interesting code and assembly along the way, so get ready for SIMD C# code, x64 SIMD assembly and tons of benchmark numbers.

However, first let’s take a look at the progression of Sep’s performance from early 0.1.0 to 0.10.0, from .NET 7.0 to .NET 9.0 and from AMD Ryzen 9 5950X (Zen 3) to 9950X (Zen 5), as I have also recently upgraded my work PC.

Sep Performance Progression

The benchmark numbers above are for the package assets CSV data and the low level parse Rows only scope, see Sep README on GitHub or code on GitHub for details on this. Note that all numbers here are single-threaded and are also shown in the table below. Note that there can be a few percentage points variation in the numbers, so for a given release Sep might see minor regressions.

The main take away is that Sep has seen incremental improvements to performance driven by both major (e.g. almost complete rewrite of internals in 0.2.0) and minor code changes. While also seeing improved performance on new .NET versions. And finally here showing improvement for going from the AMD 5950X (Zen 3) to AMD 9950X (Zen 5). Hence, this showcases how software together with hardware improvements can boost performance to the next level.

We can see Sep progressing:

• ~ 7 GB/s (0.1.0, 5950X and .NET 7.0)
• ~12 GB/s (0.3.0, 5950X and .NET 8.0)
• ~13 GB/s (0.6.0, 5950X and .NET 9.0)
• ~18 GB/s (0.9.0, 9950X and .NET 9.0)
• ~21 GB/s (0.10.0, 9950X and .NET 9.0)

This is a staggering ~3x improvement in just under 2 years since Sep was introduced June, 2023.

The improvement from 5950X w. Sep 0.9.0 to 9950X w. Sep 0.10.0 is ~1.6x which is a pretty good improvement from Zen 3 to Zen 5. Note the 9950X has a 5.7 GHz boost frequency vs 4.9 GHz for 5950X, so this alone probably explains 1.2x.

AVX-512 Code Generation and Mask Register Issues

Sep has had support for AVX-512 since 0.2.3 and back then I noted that:

different here is the use of the mask registers (k1-k8) introduced with AVX-512. However, .NET 8 does not have explicit support for these and the code generation is a bit suboptimal, given mask register are moved to normal registers each time. And then back.

I did not have direct access to an AVX-512 capable CPU then, so I could not test the performance of the AVX-512 in detail, but did verify it on the Xeon Silver 4316 which based on some quick tests showed the AVX-512 parser to be the fastest on that CPU despite the issues with the mask registers.

9950X Upgrade and AVX-512 vs AVX2 Performance

Recently, I then upgraded from an AMD 5950X (Zen 3) CPU to an AMD 9950X (Zen 5) CPU. Zen 3 does not support AVX-512, but Zen 5 does. One of the first things I did on the new CPU was of course to run the Sep benchmarks, and this showed, that Sep hit ~18 GB/s on the 9950X for the low-level parsing of CSV files. This was great and ~1.4x faster than on the 5950X. A pretty good improvement from Zen 3 to Zen 5.

However, I still wanted to compare AVX-512 to AVX2. Sep has, unofficial, support for overriding the default selected parser via an environment variable. This is also used for testing all possible parsers fully no matter which parser is selected as best. A bit surprisingly the AVX2 parser on 9950X hit ~20GB/s! That is, it was better than the AVX-512 based parser by ~10%, which is pretty significant for Sep. Hence, it would seem the mask register issue was still an issue.

Parser Code/Assembly Comparison and New AVX-512-to-256 Parser

Let’s examine the code and assembly (via Disasmo) for the AVX-512-based parser (0.9.0), a tweaked version (0.10.0), compare it to the AVX2-based parser, and finally review a new AVX-512-to-256-based parser that circumvents the mask register issue and is even faster than the AVX2-based parser, achieving ~21 GB/s as shown above.

Parse Methods

All parsers in Sep follow the same basic layout as shown below and have a single generic Parse method to support both parsing for when handling quotes (ParseColInfos) and when not (ParseColEnds). The former requires keeping track of more state, and is slightly slower.

In Sep the Parse method is marked with AggressiveInlining to ensure it is inlined, which means one can in principle go to ParseColEnds in Visual Studio with Disasmo installed and hit ALT + SHIFT + D. Unfortunately, for some reason this does not work currently unless you change the parser from a class to struct. So readers are aware if they want to follow along. See GitHub issue Empty disassembly for method with inlined generic method (used to work) for more.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
public void ParseColEnds(SepReaderState s)
{
    Parse<int, SepColEndMethods>(s);
}

public void ParseColInfos(SepReaderState s)
{
    Parse<SepColInfo, SepColInfoMethods>(s);
}

void Parse<TColInfo, TColInfoMethods>(SepReaderState s)
    where TColInfo : unmanaged
    where TColInfoMethods : ISepColInfoMethods<TColInfo>
{
    // Implementation
}

To recap, parsing in Sep is done on a span of chars (from an array) e.g. 16K and outputs a set of column end indices and row column counts for that span. This ensures parsing is done on a significant yet small enough chunk of data to fit in the CPU cache, and facilitates efficient multi-threading after too.

Parsing of the span is basically then just a loop where one or two SIMD registers (e.g. Vector256) are loaded (as unsigned 16-bit integers e.g. ushort) and converted to byte SIMD register and then compared to the special characters (e.g. \n, \r, ", ;) using SIMD compare instructions. The compare results are then converted to bit masks and each set bit in that mask is sequentially parsed after.

The interesting part here is the SIMD code and how it’s JIT’ed to machine code on .NET and how efficient that is. Below this specific code and assembly is shown for the parsers mentioned before.

SepParserAvx512PackCmpOrMoveMaskTzcnt.cs (0.9.0)

A breakdown of the below code snippet:

1. Data Loading and Packing:
   • Two 16-bit integer vectors (v0 and v1) are read from memory using unaligned reads.
   • These vectors are packed into a single byte vector using PackUnsignedSaturate, ensuring values fit within the byte range.
   • For AVX-512 this means loading two 512-bit SIMD registers each with 32 chars and then packing it to single 512-bit SIMD register with 64 bytes. This means 64 chars are handled in each loop.
2. Reordering Packed Data:
   • The packed data is interleaved, so a permutation operation (PermuteVar8x64) is applied to reorder the bytes into the correct sequence.
3. Character Comparisons:
   • The byte vector is compared against specific characters (e.g. \n, \r, ", ;) using SIMD equality operations. These comparisons identify special characters relevant to CSV parsing.
4. Combine Comparison Results:
   • The results of the comparisons are combined using logical operations.
5. Bitmask Generation and Check:
   • A MoveMask operation extracts a bitmask from the SIMD register, allowing for a quick check to skip further processing if no special characters are found.

All parsers follow the same basic approach, so this description will be omitted going forward. Note how ISA and Vec are aliases used to make the different parsers more similar which makes it easier to compare and maintain the different parsers.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
var v0 = ReadUnaligned<VecI16>(ref byteRef);
var v1 = ReadUnaligned<VecI16>(ref Add(ref byteRef, VecUI8.Count));
var packed = ISA.PackUnsignedSaturate(v0, v1);
// Pack interleaves the two vectors need to permute them back
var permuteIndices = Vec.Create(0L, 2L, 4L, 6L, 1L, 3L, 5L, 7L);
var bytes = ISA.PermuteVar8x64(packed.AsInt64(), permuteIndices).AsByte();

var nlsEq = Vec.Equals(bytes, nls);
var crsEq = Vec.Equals(bytes, crs);
var qtsEq = Vec.Equals(bytes, qts);
var spsEq = Vec.Equals(bytes, sps);

var lineEndings = nlsEq | crsEq;
var lineEndingsSeparators = spsEq | lineEndings;
var specialChars = lineEndingsSeparators | qtsEq;

// Optimize for the case of no special character
var specialCharMask = MoveMask(specialChars);
if (specialCharMask != 0u)

Assembly is shown below for a 64-bit CPU with AVX-512 support e.g. the 9950X. What is most interesting here is that each compare Vec.Equals ends up being two instructions vpcmpeqb (compare equal bytes) and vpmovm2b (move mask to byte). That is, there is a lot of going from the mask register, e.g. k1, to a normal 512-bit register, e.g. zmm5, and back again.

Note that the C# code does not deal with vector mask registers directly. This is not supported in .NET and hence it is the JIT that is responsible for code generation around this. Unfortunately, here it does not do a good job and the AVX-512 is not as fast as it could be.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
mov      edi, r9d
lea      rdi, bword ptr [r10+2*rdi]
vmovups  zmm4, zmmword ptr [rdi]
vpackuswb zmm4, zmm4, zmmword ptr [rdi+0x40]
vmovups  zmm5, zmmword ptr [reloc @RWD00]
vpermq   zmm4, zmm5, zmm4
vpcmpeqb k1, zmm4, zmm0
vpmovm2b zmm5, k1
vpcmpeqb k1, zmm4, zmm1
vpmovm2b zmm16, k1
vpcmpeqb k1, zmm4, zmm2
vpmovm2b zmm17, k1
vpcmpeqb k1, zmm4, zmm3
vpmovm2b zmm4, k1
vpternlogd zmm5, zmm4, zmm16, -2
vpord    zmm16, zmm5, zmm17
vpmovb2m k1, zmm16
kmovq    r15, k1
test     r15, r15
je       G_M000_IG03
vpmovb2m k1, zmm4
kmovq    r13, k1
lea      r12, [r15+r8]
cmp      r13, r12
je       G_M000_IG43

SepParserAvx512PackCmpOrMoveMaskTzcnt.cs (0.10.0)

To address the above code generation issues, in Sep 0.10.0 I changed the AVX-512 based parser by moving the MoveMask calls earlier to avoid the whole mask register back and forth as shown below. For other parsers, MoveMask is only called when necessary to reduce instructions in the “happy”/skip path.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
var v0 = ReadUnaligned<VecI16>(ref byteRef);
var v1 = ReadUnaligned<VecI16>(ref Add(ref byteRef, VecUI8.Count));
var packed = ISA.PackUnsignedSaturate(v0, v1);
// Pack interleaves the two vectors need to permute them back
var permuteIndices = Vec.Create(0L, 2L, 4L, 6L, 1L, 3L, 5L, 7L);
var bytes = ISA.PermuteVar8x64(packed.AsInt64(), permuteIndices).AsByte();

var nlsEq = MoveMask(Vec.Equals(bytes, nls));
var crsEq = MoveMask(Vec.Equals(bytes, crs));
var qtsEq = MoveMask(Vec.Equals(bytes, qts));
var spsEq = MoveMask(Vec.Equals(bytes, sps));

var lineEndings = nlsEq | crsEq;
var lineEndingsSeparators = spsEq | lineEndings;
var specialChars = lineEndingsSeparators | qtsEq;

// Optimize for the case of no special character
var specialCharMask = specialChars;
if (specialCharMask != 0u)

This improves the assembly for the parser quite a bit as can be seen below. Basically, less instructions. We are still going to mask register to normal register but at least only once.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
 mov      edi, r9d
 lea      rdi, bword ptr [r10+2*rdi]
 vmovups  zmm4, zmmword ptr [rdi]
 vpackuswb zmm4, zmm4, zmmword ptr [rdi+0x40]
 vmovups  zmm5, zmmword ptr [reloc @RWD00]
 vpermq   zmm4, zmm5, zmm4
 vpcmpeqb k1, zmm4, zmm0
 kmovq    r15, k1
 vpcmpeqb k1, zmm4, zmm1
 kmovq    r13, k1
 vpcmpeqb k1, zmm4, zmm2
 kmovq    r12, k1
 vpcmpeqb k1, zmm4, zmm3
 kmovq    rcx, k1
 or       r15, rcx
 or       r15, r13
 or       r12, r15
 je       SHORT G_M000_IG03
 mov      r13, rcx
 lea      rcx, [r12+r8]
 cmp      r13, rcx
 je       G_M000_IG43

SepParserAvx2PackCmpOrMoveMaskTzcnt.cs (0.10.0)

Let’s compare the AVX-512 to the AVX2 based parser. C# code is shown below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
var v0 = ReadUnaligned<VecI16>(ref byteRef);
var v1 = ReadUnaligned<VecI16>(ref Add(ref byteRef, VecUI8.Count));
var packed = ISA.PackUnsignedSaturate(v0, v1);
// Pack interleaves the two vectors need to permute them back
var bytes = ISA.Permute4x64(packed.AsInt64(), 0b_11_01_10_00).AsByte();

var nlsEq = Vec.Equals(bytes, nls);
var crsEq = Vec.Equals(bytes, crs);
var qtsEq = Vec.Equals(bytes, qts);
var spsEq = Vec.Equals(bytes, sps);

var lineEndings = nlsEq | crsEq;
var lineEndingsSeparators = spsEq | lineEndings;
var specialChars = lineEndingsSeparators | qtsEq;

// Optimize for the case of no special character
var specialCharMask = MoveMask(specialChars);
if (specialCharMask != 0u)

The assembly below is, however, clearly more straightforward as there are no mask registers involved. This explains why the AVX2 based parser is faster than the old (0.9.0) AVX-512 based parser.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
mov      edi, r9d
lea      rdi, bword ptr [r10+2*rdi]
vmovups  ymm4, ymmword ptr [rdi]
vpackuswb ymm4, ymm4, ymmword ptr [rdi+0x20]
vpermq   ymm4, ymm4, -40
vpcmpeqb ymm5, ymm4, ymm0
vpcmpeqb ymm6, ymm4, ymm1
vpcmpeqb ymm7, ymm4, ymm2
vpcmpeqb ymm4, ymm4, ymm3
vpternlogd ymm5, ymm4, ymm6, -2
vpor     ymm6, ymm5, ymm7
vpmovmskb r15d, ymm6
mov      r15d, r15d
test     r15, r15
je       SHORT G_M000_IG03
vpmovmskb r13d, ymm4
mov      r13d, r13d
lea      r12, [r15+r8]
cmp      r13, r12
je       G_M000_IG43

SepParserAvx512To256CmpOrMoveMaskTzcnt.cs (0.10.0)

Given that even the tweaked AVX-512 (0.10.0) based parser had issues with mask registers, I kept thinking that perhaps there was a more straightforward way to do this, and then after some searching and unfruitful discussions with LLMs I figured out that one could just use AVX-512 instructions for loading the chars and then convert the 16-bit to 8-bit bytes saturated as a 256-bit register, avoiding the 512-bit mask registers, by using ConvertToVector256ByteWithSaturation (vpmovuswb) as shown below. This “only” parses 32 chars at a time, but it is much simpler and avoids the mask register issue.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
var v = ReadUnaligned<VecUI16>(ref byteRef);
var bytes = ISA.ConvertToVector256ByteWithSaturation(v);

var nlsEq = Vec.Equals(bytes, nls);
var crsEq = Vec.Equals(bytes, crs);
var qtsEq = Vec.Equals(bytes, qts);
var spsEq = Vec.Equals(bytes, sps);

var lineEndings = nlsEq | crsEq;
var lineEndingsSeparators = spsEq | lineEndings;
var specialChars = lineEndingsSeparators | qtsEq;

// Optimize for the case of no special character
var specialCharMask = MoveMask(specialChars);
if (specialCharMask != 0u)

The assembly then is much simpler and direct (closer to AVX2) and not only avoids the mask register issues but also has more straightforward saturated conversion since no permutation is needed as the packed data is already in order just in the ymm4 register (that is the 256-bit part of zmm4).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
mov      edi, r9d
lea      rdi, bword ptr [r10+2*rdi]
vmovups  zmm4, zmmword ptr [rdi]
vpmovuswb zmm4, zmm4
vpcmpeqb ymm5, ymm4, ymm0
vpcmpeqb ymm6, ymm4, ymm1
vpcmpeqb ymm7, ymm4, ymm2
vpcmpeqb ymm4, ymm4, ymm3
vpternlogd ymm5, ymm4, ymm6, -2
vpor     ymm6, ymm5, ymm7
vpmovmskb r15d, ymm6
mov      r15d, r15d
test     r15, r15
je       SHORT G_M000_IG03
vpmovmskb r13d, ymm4
mov      r13d, r13d
lea      r12, [r15+r8]
cmp      r13, r12
je       G_M000_IG43

And this is what brings Sep parsing up to a staggering 21 GB/s on the 9950X! 🚀

All Parsers Benchmarks

Finally, given all the parsers available in Sep I have added a benchmark, that uses the aforementioned environment variable, to run all parsers and compare their performance on the low level row parsing to better gauge their individual performance on the same CPU. Here the AMD 9950X.

The new AVX-512-to-256 parser is the fastest parser of all hitting ~21.5 GB/s, but the Vector256/AVX2 based parsers are not far behind (about 5%). SepParserVector256NrwCmpExtMsbTzcnt is the cross-platform Vector256 based parser and it is notably now on par with the AVX2, but note how the other cross-platform Vector128 and Vector512 based parsers are not (still fast but 5-10% slower), and even worse that the Vector512 one is slower than the Vector128.

SepParserIndexOfAny is far behind, and should make it clear that any ideas that this could be used to compete with Sep are not realistic. 😉 Vector64 is not accelerated on the 9950X and therefore very slow. It’s just there for completeness.

Top Level 5950X vs 9950X Benchmarks

Finally, the table below shows the top level benchmarks for the 5950X and 9950X CPUs for the package assets and floats data.

Note how on the 9950X the one million package assets rows are parsed in just 72 ms for Sep_MT (multi-threaded Sep) compared to 119 ms on the 5950X. Or 8 GB/s on 9950X vs 4.9 GB/s on 5950X. ~8 GB/s! 🚀

Similarly, for floats Sep can parse 8 GB/s of floating point CSV data multi-threaded. ~8 GB/s! 🌪

That’s, about 1.5x-1.6x improvement, similarly to the low level benchmarks, going from 5950X to 9950X. That’s significant generational improvements to CPU performance. Kudos to AMD and TSMC.

Package Assets 5950X

Package Assets 9950X

Floats 5950X

Floats 9950X

🌟 Summary Highlights

AI generated summary highlights 😁

• 🚀 Blazing Fast Parsing: Sep 0.10.0 achieves an incredible 21 GB/s CSV parsing speed on AMD 9950X, a ~3x improvement since its first release in 2023!
• 🖥 Hardware Boost: Upgrading from AMD 5950X (Zen 3) to AMD 9950X (Zen 5) delivers a ~1.6x performance gain, thanks to AVX-512 support and higher clock speeds.
• 🧠 Smarter Parsers: The new AVX-512-to-256 parser circumvents mask register inefficiencies, outperforming AVX2 and older AVX-512 parsers, achieving ~21 GB/s!
• 📊 Cross-Platform Excellence: The Vector256 based cross-platform parser is now on par with AVX2, ensuring top-tier performance across platforms.
• 🔬 Deep Dive: Explored .NET 9.0 JIT optimizations, SIMD assembly, and parser design for CSV parsing.
• 🏆 Multi-Threaded Power: Sep parses 1 million rows in just 72 ms on the 9950X, achieving 8 GB/s for real-world CSV workloads.
• 🔧 Continuous Improvement: Incremental optimizations and hardware advancements have propelled Sep to new heights in just under 2 years.

🎉 Sep is a testament to the power of software and hardware working together to push the boundaries of performance!

That’s all!

See v0.9.0 release for all changes, the release includes a few other niceties, and Sep README on GitHub for full details. Below is a (belated) blog post focusing on the pragmatic approach used to add async support. First, however, a copy of the section on async support in Sep README to introduce this, then details on how this support was added.

Async Support

Sep supports efficient ValueTask based asynchronous reading and writing.

However, given both SepReader.Row and SepWriter.Row are ref structs, as they point to internal state and should only be used one at a time, async/await usage is only supported on C# 13.0+ as this has support for “ref and unsafe in iterators and async methods” as covered in What’s new in C# 13. Please consult details in that for limitations and constraints due to this.

Similarly, SepReader only implements IAsyncEnumerable<SepReader.Row> (and IEnumerable<SepReader.Row>) for .NET 9.0+/C# 13.0+ since then the interfaces have been annotated with allows ref struct for T.

Async support is provided on the existing SepReader and SepWriter types similar to how TextReader and TextWriter support both sync and async usage. This means you as a developer are responsible for calling async methods and using await when necessary. See below for a simple example and consult tests on GitHub for more examples.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
var text = """
           A;B;C;D;E;F
           Sep;🚀;1;1.2;0.1;0.5
           CSV;✅;2;2.2;0.2;1.5
           
           """; // Empty line at end is for line ending

using var reader = await Sep.Reader().FromTextAsync(text);
await using var writer = reader.Spec.Writer().ToText();
await foreach (var readRow in reader)
{
    await using var writeRow = writer.NewRow(readRow);
}
Assert.AreEqual(text, writer.ToString());

Note how for SepReader the FromTextAsync is suffixed with Async to indicate async creation, this is due to the reader having to read the first row of the source at creation to determine both separator and, if file has a header, column names of the header. The From*Async call then has to be awaited. After that rows can be enumerated asynchronously simply by putting await before foreach. If one forgets to do that the rows will be enumerated synchronously.

For SepWriter the usage is kind of reversed. To* methods have no Async variants, since creation is synchronous. That is, StreamWriter is created by a simple constructor call. Nothing is written until a header or row is defined and Dispose/DisposeAsync is called on the row.

For reader nothing needs to be asynchronously disposed, so using does not require await. However, for SepWriter dispose may have to write/flush data to underlying TextWriter and hence it should be using DisposeAsync, so you must use await using.

To support cancellation many methods have overloads that accept a CancellationToken like the From*Async methods for creating a SepReader or for example NewRow for SepWriter. Consult Public API Reference for full set of available methods.

Additionally, both SepReaderOptions and SepWriterOptions feature the bool AsyncContinueOnCapturedContext option that is forwarded to internal ConfigureAwait calls, see the ConfigureAwait FAQ for details on that.

Pragmatic Async Support Implementation

async/await is viral. For any async method call, however deep it may be, all methods from top to deepest async method need to be async too. This means supporting both sync and async becomes problematic as you are faced with either trying to refactor, while still needing to copy entire method chains, or copy pasting everything.

For Sep I choose the latter with a twist. Isolate IO calling methods e.g. methods using TextReader for SepReader and TextWriter for SepWriter. Then create two separate files for each of these methods: one for synchronous methods and one for asynchronous methods. These files are nearly identical, differing only by a preprocessor directive defined at the top of each file. For example:

src/Sep/SepReader.IO.Async.cs

1
//#define SYNC

src/Sep/SepReader.IO.Sync.cs

1
#define SYNC

The rest of these files are then identical, but for each method in these files #if #else #endif preprocessor switches are used to handle differences in the method signature and implementation. All async method names are then suffixed with Async to differentiate from the sync methods. For example:

1
2
3
4
5
6
7
8
9
10
11
12
#if SYNC
    internal void Initialize(in SepReaderOptions options)
#else
    internal async ValueTask InitializeAsync(in SepReaderOptions options, 
        CancellationToken cancellationToken)
#endif
    {
#if SYNC
        if (MoveNext())
#else
        if (await MoveNextAsync(cancellationToken))
#endif

This means it is easy to maintain. For consistency unit tests ensure the two files are kept in sync in face of changes by simply comparing these pairs of files and that all lines are the same except for the first line.

This approach “avoids” duplicating logic while maintaining separate implementations for sync and async operations. The shared logic is preserved by keeping the core functionality identical, with only the method signatures and specific async-related keywords differing.

By using preprocessor directives, each variant has only the relevant code for either sync or async, ensuring no unnecessary runtime checks or sync over async.

Performance

All Async methods are implemented using ValueTask to avoid the overhead of allocating Task instances if not needed (e.g. if data is already available). This means the overhead is minimal, which can be seen in benchmarks (based on in memory data in the form of StringReader) as shown below.

Sep_Async is only 1.07x slower than sync Sep at the very lowest level of simply parsing the CSV file. For any real workload the difference is neglible.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
BenchmarkDotNet v0.14.0, Windows 10 (10.0.19044.3086/21H2/November2021Update)
AMD Ryzen 9 5950X, 1 CPU, 32 logical and 16 physical cores
.NET SDK 9.0.102
  [Host]     : .NET 9.0.1 (9.0.124.61010), X64 RyuJIT AVX2
  Job-RANURT : .NET 9.0.1 (9.0.124.61010), X64 RyuJIT AVX2

Job=Job-RANURT  EnvironmentVariables=DOTNET_GCDynamicAdaptationMode=0  Runtime=.NET 9.0  
Toolchain=net90  InvocationCount=Default  IterationTime=350ms  
MaxIterationCount=15  MinIterationCount=5  WarmupCount=6  
Quotes=False  Reader=String  

| Method       | Scope | Rows    | Mean         | Ratio | MB  | MB/s    | ns/row | Allocated     | Alloc Ratio |
|------------- |------ |-------- |-------------:|------:|----:|--------:|-------:|--------------:|------------:|
| Sep______    | Row   | 50000   |     2.230 ms |  1.00 |  29 | 13088.4 |   44.6 |       1.09 KB |        1.00 |
| Sep_Async    | Row   | 50000   |     2.379 ms |  1.07 |  29 | 12264.0 |   47.6 |       1.02 KB |        0.93 |
| Sep_Unescape | Row   | 50000   |     2.305 ms |  1.03 |  29 | 12657.6 |   46.1 |       1.02 KB |        0.93 |

That’s all!

• 🎯 Remove net7.0 target
• ✨ SepWriter.Col: Replace StringBuilder with ArrayPool array and DefaultInterpolatedStringHandler

See v0.8.0 release for all changes and Sep README on GitHub for full details. Below is a quick (belated) blog post to explain the changes a bit.

SepWriter vs TextWriter

SepWriter hasn’t gotten as much attention as SepReader here, which is partly intentional, as SepWriter is not so much about performance and speed but more about convenience, ease of use and change. And not much has changed about that since Sep was introduced. If you want the best speed for writing you would be better off simply using TextWriter directly (if done correctly).

Let’s do a quick code comparison of SepWriter and TextWriter. Given:

1
2
3
4
const string ColNameA = "A";
const string ColNameB = "B";

ReadOnlySpan<int> values = [1, 2, 3];

we want to write some multiple of the values to csv for each column. With Sep this can be done like below. The main take away here is that with Sep you do not have to separate the writing of the header (column name) and the values.

1
2
3
4
5
6
7
8
using var sepWriter = Sep.Default.Writer().ToText();
foreach (var v in values)
{
    using var row = sepWriter.NewRow();
    row[ColNameA].Format(v * 10);
    row[ColNameB].Format(v * 100);
}
Console.WriteLine(sepWriter.ToString());

One way to do this with StringWriter (aka TextWriter) is shown below. While this clearly is longer, the other issue is how you have two separate parts for first writing the header and then writing the rows. Not a big issue here but when you have many columns keeping things in sync can be a challenge and a known source of dev churn.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
const char Separator = ';';
using var textWriter = new StringWriter();
// Header
textWriter.Write(ColNameA);
textWriter.Write(Separator);
textWriter.Write(ColNameB);
textWriter.WriteLine();
// Rows
foreach (var v in values)
{
    textWriter.Write(v * 10);
    textWriter.Write(Separator);
    textWriter.Write(v * 100);
    textWriter.WriteLine();
}
Console.WriteLine(textWriter.ToString());

SepWriter.Col: StringBuilder Issue

The above TextWriter code is basically what Sep does under the hood. However, for each column (e.g. var col = row[ColNameA];) Sep would store each column value as a StringBuilder until the completion of a row and calling Dispose() on it at which point the contents of StringBuilder is written to the underlying TextWriter that SepWriter works over. In this way Sep could utilize all the StringBuilder functionality to support Format (e.g. ISpanFormattable) and similar. Additionally, Sep would use a pool of StringBuilders to reduce repeated allocations.

StringBuilder does have an underlying issue, though, in that it is basically implemented as a linked list of StringBuilders, which means in order to write all the contents of it to TextWriter, without creating a string, one would have to enumerate the chunks of it like:

1
2
3
4
foreach (var chunk in sb.GetChunks())
{
    _writer.Write(chunk.Span);
}

Since, long columns are rare it is similarly rare for there being multiple chunks. Hence, the enumeration causes a bit of a performance hit.

SepWriter.Col: Replace StringBuilder with ArrayPool array and DefaultInterpolatedStringHandler

Performance is a feature, and while not the top priority for SepWriter, 0.8.0 addresses this issue by swapping out the internal StringBuilder with a char[] from ArrayPool. To implement formatting Sep then relies on DefaultInterpolatedStringHandler.

However, this doesn’t have public APIs allowing for using and managing arrays from the ArrayPool. I was then faced with a choice of either copying the entire implementation of DefaultInterpolatedStringHandler or finding another way. That other way was to use the UnsafeAccessor attribute to access the internal state of DefaultInterpolatedStringHandler, as shown below, and reuse the array from ArrayPool. This is a bit of a hack, but it works and is fast.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// Avoid recreating DefaultInterpolatedStringHandler while being
// able to reuse array from ArrayPool by using UnsafeAccessor to
// access internal state of this. This works fine for net8.0 and
// net9.0 but there are no guarantees if this could change in the
// future, if so consider using #if NET10_0_OR_GREATER or similar to
// address any changes or consider then copying the entire
// DefaultInterpolatedStringHandler source code and adopt for needs.
 
[MethodImpl(MethodImplOptions.AggressiveInlining)]
[UnsafeAccessor(UnsafeAccessorKind.Field, Name = "_arrayToReturnToPool")]
static extern ref char[]? ArrayToReturnToPool(ref DefaultInterpolatedStringHandler handler);
 
[MethodImpl(MethodImplOptions.AggressiveInlining)]
[UnsafeAccessor(UnsafeAccessorKind.Field, Name = "_pos")]
static extern ref int Position(ref DefaultInterpolatedStringHandler handler);

The downside is that UnsafeAccessor is only supported on net8.0 and above. Given net7.0 is no longer supported I decided it was time to drop support for it.

I don’t have detailed benchmarks here, but the end result for SepWriter is that for a given simple case of writing multiple short columns SepWriter is 10-15% faster while still having zero allocations after warmup/first rows. Additionally, code is simpler, even with the UnsafeAccessor code.

For more details take a look at the pull request SepWriter.Col: Replace StringBuilder with ArrayPool array and DefaultInterpolatedStringHandler.

That’s all!

The script targets scenarios where you want to list all PRs and filter them based on criteria such as creation date and the author’s display name (for example, matching initials or part of a full name). That is, it gets all pull requests for a project since a given date, filters them, and accumulates all to finally display them sorted by creation date.

Why Use This Script?

• Retrieves all projects in your organization.
• Pulls all PRs per project since a given date using the Azure DevOps REST API and pagination.
• Filters PRs by checking the createdBy.displayName field.
• Adds the project name to the PR object for easier reporting.
• Outputs the results in a neatly formatted table.

Prerequisites

• Azure DevOps Personal Access Token (PAT): You need a PAT with at least Code (read) permissions. Generate one from your Azure DevOps account settings.
• PowerShell Environment: This script uses basic PowerShell commands available on modern Windows systems.
• Basic Understanding of REST APIs: Familiarity with REST API concepts can help in customizing the script further.

The Script

Below is the complete PowerShell script. Update the variable values (e.g., organization name, PAT, author initials, full name) to match your environment.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# Define variables
$organization = "your-org-name"          # Replace with your Azure DevOps organization name
$personalAccessToken = "your-pat-token"  # Replace with your Azure DevOps PAT
$top = 400                               # Number of PRs to retrieve per request
$authorInitials = "your-initials"        # Substring to match (e.g., initials)
$fullName = "your-name"                  # Full name to match
$lastYearDate = (Get-Date).AddYears(-1).ToString("yyyy-MM-ddTHH:mm:ssZ")

# Encode PAT for authentication
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$personalAccessToken"))

# Function to retrieve PRs for a given project
function Get-PullRequests {
    param (
        [string]$projectName
    )
    $skip = 0
    $prs = @()
    do {
        # Construct the API URL with time range filters
        $prUrl = "https://dev.azure.com/$organization/$projectName/_apis/git/pullrequests?searchCriteria.status=all&`$top=$top&`$skip=$skip&searchCriteria.minTime=$lastYearDate&searchCriteria.queryTimeRangeType=Created&api-version=7.1"
        $response = Invoke-RestMethod -Uri $prUrl -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -Method Get
        $prs += $response.value
        $skip += $top
    } while ($response.value.Count -eq $top)  # Continue if the number of PRs retrieved equals $top
    return $prs
}

# Get all projects in the organization
$projectsUrl = "https://dev.azure.com/$organization/_apis/projects?api-version=7.1-preview.4"
$projects = Invoke-RestMethod -Uri $projectsUrl -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -Method Get

$allPullRequests = @()

foreach ($project in $projects.value) {
    $projectName = $project.name
    $prs = Get-PullRequests -projectName $projectName

    $filteredPRs = $prs | Where-Object {
        ( $_.createdBy.displayName -match $authorInitials -or $_.createdBy.displayName -match $fullName )
    } | ForEach-Object {
        # Manually add the project name (since it's not part of the PR object)
        $_ | Add-Member -NotePropertyName "Project" -NotePropertyValue $projectName -PassThru
    }
    $allPullRequests += $filteredPRs
}

# Sort the results by creationDate and output desired columns
$allPullRequests |
    Sort-Object creationDate |
    Select-Object Project, @{Name="Repository"; Expression={ $_.repository.name }}, pullRequestId, title, creationDate, status |
    Format-Table -AutoSize

I have verified this script works for my needs, but use at your own peril. Output may look something like.

1
2
3
4
5
6
Project     Repository pullRequestId title           creationDate    status
-------     ---------- ------------- -----           ------------    ------
ProjectName       repo         4001 PR title 2024-01-01T11:00:00 completed
ProjectName       repo         4002 PR title 2024-01-02T12:00:00 completed
ProjectName       repo         4003 PR title 2024-01-03T13:00:00 completed
ProjectName       repo         4004 PR title 2024-01-04T14:00:00 completed

That’s all!

• 🎁 Add SepWriterOptions.Escape for escape support
• 🎛️ Add SepWriterOptions.DisableColCountCheck/ColNotSetOption

See v0.7.0 release for all changes and Sep README on GitHub for full details. Below I’ll briefly go over these changes. Escape support is one of the last major features that was missing in Sep compared to other libraries.

Escape Support

Sep now supports escaping by the Escape property on SepWriterOptions which can be set like:

1
2
using var writer = Sep.Writer(o => 
    o with { Escape = true }).ToText();

The result of escaping is shown below in comparison to other popular CSV libraries. All basically do the same, except CsvHelper which also escapes spaces despite this not being necessary.

Separator/delimiter is set to semi-colon ; (default for Sep)

· (middle dot) is whitespace to make this visible

\r, \n are carriage return and line feed special characters to make these visible

¹ Sep with Escape = true in SepWriterOptions

SepWriterOptions.DisableColCountCheck/ColNotSetOption

These new options are for allowing writing CSV files with different columns per row and defining how to handle columns not set. Below is just one example of how this could be used.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
var options = new SepWriterOptions
{
    WriteHeader = false,
    DisableColCountCheck = true,
    ColNotSetOption = SepColNotSetOption.Skip,
};
using var writer = options.ToText();
{
    using var row = writer.NewRow();
    row["A"].Set("R1C1");
    row["B"].Set("R1C2");

}
{
    using var row = writer.NewRow();
    row[0].Set("R2C1");
    row[1].Set("R2C2");
    row[2].Set("R2C3");
    row[3].Set("R2C4");
}
{
    using var row = writer.NewRow();
    row["A"].Set("R3C1");
    row[2].Set("R3C3");
    row[1].Set("R3C2");
}
var expected =
@"R1C1;R1C2
R2C1;R2C2;R2C3;R2C4
R3C1;R3C2;R3C3
";
Assert.AreEqual(expected, writer.ToString());

Note how that each row has different number of columns and that any column not set is skipped. There is also an option for writing an empty column if not set, for example.

For more examples see tests on GitHub.

That’s all!

• 🐛 Bug fix to SepWriter when selecting multiple columns by indices
• ✂️ SepReader trim support
• 🤖 .NET 9 ready
• 🚀 Updated and new benchmarks incl. Apple M1

See v0.6.0 release for all changes and Sep README on GitHub for full details. Below I’ll go over the notable changes briefly. Keep reading for perf numbers!

Bug Fix to SepWriter

Yet another reminder that great code coverage (Sep has ~100% now) does not preclude any bugs… a functioning brain should. However, sometimes a usually reasonably well functioning brain has a day off and in that case wrote a test wrong which meant a bug snuck into SepWriter.Row when selecting multiple columns by indices e.g. for:

1
var cols = row[3, 2];

one would expect columns at indices 3 and 2 to be selected, but instead indices 0 and 1 were selected. A simple mistake fixed by using colIndices[i] instead of just i. I would guess no one have actually used this, or hope. At least there’s been no such usage at work.

Trim Support

Sep now supports trimming by the SepTrim flags enum, which has two options as documented in the code. To enable trimming set the option like below:

1
2
using var reader = Sep.Reader(o => 
    o with { Trim = SepTrim.All }).FromText(text);

Below the result of both trimming and unescaping is shown in comparison to CsvHelper. Note unescaping is enabled for all results shown. It is possible to trim without unescaping too, of course.

As can be seen Sep supports a simple principle of trimming before and after unescaping with trimming before unescaping being important for unescaping if there is a starting quote after spaces.

· (middle dot) is whitespace to make this visible

¹ CsvHelper with TrimOptions.Trim | TrimOptions.InsideQuotes

² Sep with SepTrim.All = SepTrim.Outer | SepTrim.AfterUnescape in SepReaderOptions

Trimming has a cost, of course, benchmarks below show this for AMD Ryzen 9 5950X when accessing the column Span for the package assets benchmark where most (but not all) columns have been prefixed and suffixed with ·"·, which then needs to be removed. For details on benchmarks see Sep on GitHub. As the numbers show Sep is about 11.20 / 1.44 = 7.78x to 11.07 / 1.69 = 6.54x faster than CsvHelper for this scenario. Sylvan does not appear to support trimming.

1
2
3
4
5
6
7
8
| Method                     | Scope | Rows  | Mean      | Ratio | MB | MB/s   | ns/row | Allocated | Alloc Ratio |
|--------------------------- |------ |------ |----------:|------:|---:|-------:|-------:|----------:|------------:|
| Sep_                       | Cols  | 50000 |  8.599 ms |  1.00 | 41 | 4857.5 |  172.0 |   1.04 KB |        1.00 |
| Sep_Trim                   | Cols  | 50000 | 12.402 ms |  1.44 | 41 | 3368.0 |  248.0 |   1.05 KB |        1.01 |
| Sep_TrimUnescape           | Cols  | 50000 | 13.201 ms |  1.54 | 41 | 3164.1 |  264.0 |   1.06 KB |        1.02 |
| Sep_TrimUnescapeTrim       | Cols  | 50000 | 14.568 ms |  1.69 | 41 | 2867.3 |  291.4 |   1.07 KB |        1.02 |
| CsvHelper_TrimUnescape     | Cols  | 50000 | 96.272 ms | 11.20 | 41 |  433.9 | 1925.4 | 451.52 KB |      432.51 |
| CsvHelper_TrimUnescapeTrim | Cols  | 50000 | 95.183 ms | 11.07 | 41 |  438.8 | 1903.7 | 445.86 KB |      427.09 |

These benchmarks were run using .NET 9, but there is no big difference to .NET 8, which brings us to Sep and .NET 9.

.NET 9 Ready

Sep now uses the latest .NET 9 SDK for development and besides still targeting net7.0 and net8.0, it now also targets net9.0. The main reason being to add allows ref struct annotations and support params ReadOnlySpan<>.

The former means SepReader now for .NET 9+ actually implements IEnumerable<SepReader.Row>. However, this isn’t particularly useful yet since .NET apparently hasn’t updated LINQ extension methods to have allows ref struct for TSource in any such methods. Someday perhaps. 🤷

The latter means for most indexing operations params is supported and one can write e.g. row[1, 2, 3] or row["A", "B", "C"]. Straight-forward and without allocation since the params is ReadOnlySpan<int> or ReadOnlySpan<string> and backed by stack allocated storage.

For more details on the API and usage, please refer to the README.

Updated and New Benchmarks

As part of updating for .NET 9 all benchmarks have been re-run and a new set of benchmark “machines” has been used as listed below. (Virtual) below means this machine is actually a GitHub CI agent machine, hence, it is subject to noisy neighbors and is only a subset of the cores of any full the CPU.

• AMD EPYC 7763 (Virtual) X64 Platform Information

  1
  2
  OS=Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores
  

• AMD Ryzen 7 PRO 7840U (Laptop on battery) X64 Platform Information

  1
  2
  3
  OS=Windows 11 (10.0.22631.4460/23H2/2023Update/SunValley3)
  AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics, 
  1 CPU, 16 logical and 8 physical cores
  

• AMD 5950X (Desktop) X64 Platform Information

  1
  2
  OS=Windows 10 (10.0.19044.2846/21H2/November2021Update)
  AMD Ryzen 9 5950X, 1 CPU, 32 logical and 16 physical cores
  

• Apple M1 (Virtual) ARM64 Platform Information

  1
  2
  OS=macOS Sonoma 14.7.1 (23H222) [Darwin 23.6.0]
  Apple M1 (Virtual), 1 CPU, 3 logical and 3 physical cores
  

This means the previous Neoverse M1 ARM64 processor benchmarks have been replaced with the Apple M1 processor. Results for this on the floats benchmark show Sep is ~3x-7x faster than others from low level to top level, as seen below. At the lowest level of just parsing the CSV e.g. the rows, Sep hits ~5 GB/s vs around 1 GB/s for others.

Apple M1 Floats Benchmarks

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| Method    | Scope  | Rows  | Mean       | Ratio | MB | MB/s   | ns/row | Allocated   | Alloc Ratio |
|---------- |------- |------ |-----------:|------:|---:|-------:|-------:|------------:|------------:|
| Sep______ | Row    | 25000 |   3.887 ms |  1.00 | 20 | 5215.5 |  155.5 |      1.2 KB |        1.00 |
| Sylvan___ | Row    | 25000 |  17.956 ms |  4.62 | 20 | 1129.0 |  718.2 |    10.62 KB |        8.87 |
| ReadLine_ | Row    | 25000 |  14.074 ms |  3.62 | 20 | 1440.4 |  563.0 | 73489.65 KB |   61,381.24 |
| CsvHelper | Row    | 25000 |  27.741 ms |  7.14 | 20 |  730.8 | 1109.6 |    20.28 KB |       16.94 |
|           |        |       |            |       |    |        |        |             |             |
| Sep______ | Cols   | 25000 |   4.726 ms |  1.00 | 20 | 4289.0 |  189.1 |      1.2 KB |        1.00 |
| Sylvan___ | Cols   | 25000 |  20.241 ms |  4.28 | 20 | 1001.5 |  809.6 |    10.62 KB |        8.84 |
| ReadLine_ | Cols   | 25000 |  14.976 ms |  3.17 | 20 | 1353.6 |  599.0 | 73489.65 KB |   61,181.63 |
| CsvHelper | Cols   | 25000 |  29.842 ms |  6.31 | 20 |  679.3 | 1193.7 |  21340.5 KB |   17,766.40 |
|           |        |       |            |       |    |        |        |             |             |
| Sep______ | Floats | 25000 |  24.511 ms |  1.00 | 20 |  827.1 |  980.4 |     8.34 KB |        1.00 |
| Sep_MT___ | Floats | 25000 |   9.422 ms |  0.38 | 20 | 2151.5 |  376.9 |    79.89 KB |        9.58 |
| Sylvan___ | Floats | 25000 |  69.902 ms |  2.85 | 20 |  290.0 | 2796.1 |    18.57 KB |        2.23 |
| ReadLine_ | Floats | 25000 |  79.015 ms |  3.22 | 20 |  256.6 | 3160.6 |  73493.2 KB |    8,816.43 |
| CsvHelper | Floats | 25000 | 104.811 ms |  4.28 | 20 |  193.4 | 4192.4 | 22063.34 KB |    2,646.77 |

.NET 9 Performance and DATAS Issue

If you are wondering whether .NET 9 provides any significant performance improvements over .NET 8 for Sep, then no. Some minor within 5-10% improvements have been observed, but also minor regressions. This is expected as Sep has been thoroughly optimized to get as good as possible machine code as possible, so while JIT improvements can improve this, there is not much left on the table.

However, as detailed in .NET 8.0.10 vs 9.0.0 RC2 GC Server Performance Regression in Sep (CSV Parser) Benchmark (due to DATAS default) the GC has switched to enable DATAS (Dynamically Adapting To Application Sizes) by default when using Server GC in .NET 9.

Generally, this means the GC is more aggressive with regards to running garbage collections. However, for bursty workloads like Sep’s CSV parsing and the package assets benchmark where 1 million instances of a parsed package asset rows is accumulated, this can hurt performance by up to 1.7x (for parallel enumeration) in my testing. Hence, for Sep benchmarks using GC Server mode, DATAS has been disabled, and I would recommend doing the same if you have lots of RAM, your own machine and are running machine learning pipelines like we are.

That’s all!