Hi,
I'm looking for an algorithm for compressing a large array of floats in memory.
An example of such an array would be the following:
So basically, there's some structure to the array, with numbers often being close to one another.
It's not a problem if the numbers will change slightly after compressing and decompressing; a loss of data is acceptable.
For this reason, I think the best approach would be to convert to integers, for example by multiplying by a constant like 64, and then compressing the array of integers. Just to give you an idea, here's what the corresponding array of integers would be like:
Now, I've found a library that seems to do this, namely this one: https://github.com/lemire/FastPFor
The problem here though is that this library is huge.
I'm working in Visual Studio 2013 and there seem to be something like 50 header and source files, which kind of seems like overkill.
I'm really just looking for a SIMD compression algorithm for this type of arrays of ints.
Is anyone by any chance familiar with this library?
Can I possibly just use some of the files/functions?
Unfortunately I can't seem to find any sort of description, but maybe I'm missing something.
Or does anyone possibly know of different code for compressing such arrays that's a bit more to the point?
I'm looking for an algorithm for compressing a large array of floats in memory.
An example of such an array would be the following:
Code:
5.27002
5.25617
5.25617
5.19152
5.19152
5.22103
4.09978
4.09179
4.09998
4.14264
4.14252
4.14252
4.09179
4.09998
4.14222
4.14247
4.14247
4.0922
4.14264
4.14285
4.14222
3.53468
3.51673
3.51674
3.46399
3.46399
3.46409
3.04601
3.12485
3.12252
3.12525
3.04169
3.12511
3.12251
3.12525
3.00304
3.04169
3.12511
3.00299
3.12511
3.12489
3.00299
2.20897
2.20892
2.07773
2.07773
2.20884
2.04448
2.20884
2.20423
2.20901
2.20423
2.07785
2.20897
2.04503
2.04448
2.20939
2.44582
2.43371
2.44664
2.43075
2.44641
2.43371
3.2917
3.29171
3.29211
3.29011
3.29193
3.29427
3.99966
4.07616
4.00627
4.00189
4.05059
4.07456
6.54268
6.53894
5.41779
5.42332
5.43506
5.43517
6.19722
6.19727
6.84978
7.04686
6.62073
7.04653
7.19557
6.84925
8.07813
8.08263
7.9989
8.09772
8.21839
8.09746
9.0201
9.2143
9.01947
9.07471
9.02349
9.01945
8.06667
8.06686
8.06653
8.14854
8.60916
8.0893
8.14899
8.92604
8.17176
8.06653
8.09028
8.60954
8.3946
8.55055
9.36687
8.87853
8.39398
8.45723
8.45722
8.39404
9.59805
8.55085
8.39401
9.3673
10.8436
9.73485
9.56633
9.52387
9.56614
10.6603
9.56627
9.52302
9.56261
10.6566
9.69177
9.73446
11.3881
11.2683
11.3876
10.9609
11.0024
10.9608
10.9769
11.0035
10.961
10.9765
10.8047
10.805
10.5959
10.8278
10.961
11.7564
11.8413
12.1929
11.8419
11.8085
11.8553
12.3674
11.8557
11.8557
12.3685
11.8557
11.8086
13.0753
13.0645
13.1107
12.543
13.1108
12.5432
13.1152
12.5515
12.9866
13.1133
13.0653
13.075
14.1488
14.9883
14.1484
14.9924
14.9921
14.1489
14.191
14.5116
14.5121
14.1759
13.6366
13.6369
13.9762
13.6246
13.9843
13.6051
13.9843
13.6057
11.6981
14.0951
11.6055
13.5688
13.5741
12.1942
14.0909
13.3331
13.3416
10.8083
14.3746
14.3786
11.9277
8.7986
11.7776
5.91443
11.9224
9.70141
11.7953
11.8016
11.8103
9.69659
8.78159
9.80058
13.7538
13.7539
5.98728
13.75
6.00621
13.8827
2.5328
13.762
13.7618
13.8786
13.7578
13.8756
13.6536
13.7136
4.19574
13.6515
13.4455
13.4455
13.567
13.6492
13.6536
13.7092
7.13406
7.11044
8.22532
10.4324
8.82452
9.37728
9.36628
9.36283
7.43251
8.20945
8.82081
9.3762
8.85884
10.4432
0
0
0
0
0
0
0
0
0
0
0
7.42694
7.44264
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0It's not a problem if the numbers will change slightly after compressing and decompressing; a loss of data is acceptable.
For this reason, I think the best approach would be to convert to integers, for example by multiplying by a constant like 64, and then compressing the array of integers. Just to give you an idea, here's what the corresponding array of integers would be like:
Code:
337
336
336
332
332
334
262
262
262
265
265
265
262
262
265
265
265
262
265
265
265
226
225
225
222
222
222
195
200
200
200
195
200
200
200
192
195
200
192
200
200
192
141
141
133
133
141
131
141
141
141
141
133
141
131
131
141
157
156
157
156
157
156
211
211
211
211
211
211
256
261
256
256
259
261
419
418
347
347
348
348
397
397
438
451
424
451
461
438
517
517
512
518
526
518
577
590
577
581
578
577
516
516
516
522
551
518
522
571
523
516
518
551
537
547
599
568
537
541
541
537
614
547
537
600
694
623
612
610
612
682
612
609
612
682
620
623
729
721
729
701
704
701
703
704
702
702
692
692
678
693
702
752
758
780
758
756
759
792
759
759
792
759
756
837
836
839
803
839
803
839
803
831
839
836
837
906
959
905
960
959
906
908
929
929
907
873
873
894
872
895
871
895
871
749
902
743
868
869
780
902
853
854
692
920
920
763
563
754
379
763
621
755
755
756
621
562
627
880
880
383
880
384
888
162
881
881
888
880
888
874
878
269
874
861
861
868
874
874
877
457
455
526
668
565
600
599
599
476
525
565
600
567
668
0
0
0
0
0
0
0
0
0
0
0
475
476
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0The problem here though is that this library is huge.
I'm working in Visual Studio 2013 and there seem to be something like 50 header and source files, which kind of seems like overkill.
I'm really just looking for a SIMD compression algorithm for this type of arrays of ints.
Is anyone by any chance familiar with this library?
Can I possibly just use some of the files/functions?
Unfortunately I can't seem to find any sort of description, but maybe I'm missing something.
Or does anyone possibly know of different code for compressing such arrays that's a bit more to the point?