-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
1296 lines (1179 loc) · 69.6 KB
/
Copy pathindex.html
File metadata and controls
1296 lines (1179 loc) · 69.6 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>OpenCrawling | Enterprise Ingestion & Security Framework for AI & Search</title>
<!-- SEO Meta Tags -->
<meta name="description" content="OpenCrawling is a high-performance enterprise data ingestion and security mapping framework. Built on Java 25, Spring Boot 4, Apache Kafka, Apache Tika, Redis, and Ollama.">
<meta name="keywords" content="opencrawling, enterprise ingestion, rag, vector search, acl synchronization, spring boot, java 25, virtual threads, pgvector, ollama, apache kafka, apache tika, redis">
<meta name="author" content="OpenCrawling GitHub Organization">
<link rel="canonical" href="https://opencrawling.org/">
<!-- JSON-LD Structured Data for SEO -->
<script type="application/ld+json">
[
{
"@context": "https://schema.org",
"@type": "Organization",
"name": "OpenCrawling",
"url": "https://opencrawling.org/",
"logo": "https://opencrawling.org/assets/logo-without-text-200px.png",
"sameAs": [
"https://github.com/OpenCrawling/opencrawling"
],
"description": "OpenCrawling provides open-source enterprise data ingestion, vector search, and ACL security mapping for AI agents."
},
{
"@context": "https://schema.org",
"@type": "WebSite",
"name": "OpenCrawling",
"url": "https://opencrawling.org/",
"potentialAction": {
"@type": "SearchAction",
"target": "https://opencrawling.org/?q={search_term_string}",
"query-input": "required name=search_term_string"
}
},
{
"@context": "https://schema.org",
"@type": "SoftwareApplication",
"name": "OpenCrawling",
"operatingSystem": "Cross-platform",
"applicationCategory": "DeveloperApplication",
"description": "High-performance enterprise data ingestion and security mapping framework bridging siloed content stores to AI and vector search infrastructure.",
"offers": {
"@type": "Offer",
"price": "0",
"priceCurrency": "USD"
},
"license": "https://github.com/OpenCrawling/opencrawling/blob/main/LICENSE",
"publishingPrinciples": "https://github.com/opencrawling/open-ingestion-standard"
}
]
</script>
<!-- Open Graph / Facebook -->
<meta property="og:type" content="website">
<meta property="og:url" content="https://opencrawling.org/">
<meta property="og:title" content="OpenCrawling | Enterprise Ingestion & Security Framework">
<meta property="og:description" content="High-Performance Enterprise Ingestion and Security Framework built with Java 25, Spring Boot 4, Apache Kafka, Apache Tika, Redis, and Ollama.">
<meta property="og:image" content="https://opencrawling.org/assets/logo-without-text-200px.png">
<!-- Twitter -->
<meta property="twitter:card" content="summary_large_image">
<meta property="twitter:url" content="https://opencrawling.org/">
<meta property="twitter:title" content="OpenCrawling | Enterprise Ingestion & Security Framework">
<meta property="twitter:description" content="High-Performance Enterprise Data Ingestion, Federation & Security Framework for AI and Search.">
<meta property="twitter:image" content="https://opencrawling.org/assets/logo-without-text-200px.png">
<!-- SVG Favicon -->
<link rel="icon" type="image/svg+xml" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 24 24' fill='none' stroke='%237C3AED' stroke-width='2' stroke-linecap='round' stroke-linejoin='round'%3E%3Cpath d='M12 2L2 7l10 5 10-5-10-5zM2 17l10 5 10-5M2 12l10 5 10-5'/%3E%3C/svg%3E">
<!-- Google Fonts -->
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Outfit:wght@300;400;500;600;700;800&family=Space+Grotesk:wght@400;500;600;700&family=Fira+Code:wght@400;500&display=swap" rel="stylesheet">
<!-- CSS Stylesheets -->
<link rel="stylesheet" href="css/styles.css">
</head>
<body>
<!-- Background Decorative Gradients -->
<div class="glow-bg-grid" aria-hidden="true"></div>
<div class="glow-blob glow-blob-1" aria-hidden="true"></div>
<div class="glow-blob glow-blob-2" aria-hidden="true"></div>
<!-- Header / Navigation -->
<header class="site-header">
<div class="header-container">
<a href="#" class="brand" id="nav-brand">
<img src="assets/logo-without-text-200px.png" alt="OpenCrawling Logo" class="brand-logo">
<span class="brand-name">OpenCrawling</span>
</a>
<nav class="main-nav" aria-label="Main Navigation">
<ul>
<li><a href="#features">Features</a></li>
<li><a href="sdk.html">Java SDK</a></li>
<li><a href="#standard">Open Standard</a></li>
<li><a href="#getting-started">Quick Start</a></li>
<li><a href="mcp.html">MCP Server</a></li>
<li><a href="observability.html">Observability</a></li>
<li><a href="#blog">Blog</a></li>
<li><a href="comparison.html">Compare</a></li>
<li><a href="team.html">Team</a></li>
</ul>
</nav>
<div class="header-actions">
<button id="theme-toggle" class="theme-toggle-btn" aria-label="Toggle theme">
<svg class="sun-icon" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="4"/><path d="M12 2v2M12 20v2M4.93 4.93l1.41 1.41M17.66 17.66l1.41 1.41M2 12h2M20 12h2M6.34 17.66l-1.41 1.41M19.07 4.93l-1.41 1.41"/></svg>
<svg class="moon-icon" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 3a6 6 0 0 0 9 9 9 9 0 1 1-9-9Z"/></svg>
</button>
<a href="https://github.com/OpenCrawling/opencrawling" target="_blank" rel="noopener noreferrer" class="github-btn" id="github-link">
<svg class="github-icon" viewBox="0 0 16 16" fill="currentColor" aria-hidden="true" width="18" height="18"><path d="M8 0c4.42 0 8 3.58 8 8a8.013 8.013 0 0 1-5.45 7.59c-.4.08-.55-.17-.55-.38 0-.27.01-1.13.01-2.2 0-.75-.25-1.23-.54-1.48 1.78-.2 3.65-.88 3.65-3.95 0-.88-.31-1.59-.82-2.15.08-.2.36-1.02-.08-2.12 0 0-.67-.22-2.2.82-.64-.18-1.32-.27-2-.27-.68 0-1.36.09-2 .27-1.53-1.03-2.2-.82-2.2-.82-.44 1.1-.16 1.92-.08 2.12-.51.56-.82 1.28-.82 2.15 0 3.06 1.86 3.75 3.64 3.95-.23.2-.44.55-.51 1.07-.46.21-1.61.55-2.33-.66-.15-.24-.6-.83-1.23-.82-.67.01-.27.38.01.53.34.19.73.9.82 1.13.16.45.68 1.35 3.12.88.01.64.01 1.25.01 1.42 0 .21-.15.47-.55.38A8.006 8.006 0 0 1 0 8c0-4.42 3.58-8 8-8z"/></svg>
<span>Star</span>
</a>
<button id="mobile-menu-toggle" class="mobile-menu-toggle" aria-label="Toggle Navigation" aria-expanded="false">
<span></span>
<span></span>
<span></span>
</button>
</div>
</div>
</header>
<!-- Hero Section -->
<section class="hero-section">
<div class="hero-container">
<div class="hero-content">
<div class="badge-container">
<span class="animated-badge">Enterprise Data Ingestion Platform</span>
</div>
<h1 class="hero-title">
Securely connecting <br>
<span class="gradient-text">siloed content</span> to AI pipelines.
</h1>
<p class="hero-subtitle">
OpenCrawling® is a high-performance enterprise data ingestion and security mapping framework. Powered by <strong>Java 25</strong> and <strong>Spring Boot 4</strong>, it orchestrates secure pipelines using <strong>Apache Tika</strong> for text extraction, <strong>Apache Kafka</strong> for queueing, <strong>Apache Ozone</strong> & S3 for distributed payload storage, <strong>Redis</strong> for caching, and <strong>Ollama</strong> for embeddings while preserving document permissions (ACLs).
</p>
<div class="hero-actions">
<a href="#getting-started" class="btn btn-primary">Get Started</a>
<a href="#simulator" class="btn btn-secondary">Try Live Simulator</a>
</div>
</div>
<div class="hero-visual">
<div class="terminal-card">
<div class="terminal-header">
<span class="dot red"></span>
<span class="dot yellow"></span>
<span class="dot green"></span>
<span class="terminal-title">bash - docker-setup</span>
</div>
<div class="terminal-body">
<p class="line comment"># Spin up PostgreSQL + pgvector, Redis, Ollama, Kafka</p>
<p class="line"><span class="cmd">git clone</span> https://github.com/OpenCrawling/opencrawling.git</p>
<p class="line"><span class="cmd">cd</span> opencrawling</p>
<p class="line"><span class="cmd">docker compose</span> up -d</p>
<p class="line output">Creating network "opencrawling_default" ... Created</p>
<p class="line output">Creating container opencrawling-postgres ... Running</p>
<p class="line output">Creating container opencrawling-redis ... Running</p>
<p class="line output">Creating container opencrawling-kafka ... Running</p>
<p class="line output">Creating container opencrawling-ollama ... Running</p>
<p class="line success">✔ All services started successfully on local ports.</p>
</div>
</div>
</div>
</div>
</section>
<!-- Key Metrics / Stats Section -->
<section class="stats-section">
<div class="stats-container">
<div class="stat-card">
<span class="stat-num">Java 25</span>
<span class="stat-desc">Virtual Threads & Structured Concurrency</span>
</div>
<div class="stat-card">
<span class="stat-num">100%</span>
<span class="stat-desc">ACL Security & Permission Mapping</span>
</div>
<div class="stat-card">
<span class="stat-num">Event-Driven</span>
<span class="stat-desc">Apache Kafka Document Decoupling</span>
</div>
<div class="stat-card">
<span class="stat-num">Scale-Out</span>
<span class="stat-desc">Decoupled Embedding Microservice</span>
</div>
</div>
</section>
<!-- Features Section -->
<section id="features" class="features-section">
<div class="section-header">
<h2 class="section-title">Designed for Enterprise Security & Scale</h2>
<p class="section-subtitle">Going far beyond simple scraping, OpenCrawling bridges data stores and vector indexes securely.</p>
</div>
<div class="features-grid">
<div class="feature-card">
<div class="feature-icon-wrapper">
<svg class="feature-icon" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 15v2m-6 4h12a2 2 0 002-2v-6a2 2 0 00-2-2H6a2 2 0 00-2 2v6a2 2 0 002 2zm10-10V7a4 4 0 00-8 0v4h8z"/>
</svg>
</div>
<h3>Enterprise Security First</h3>
<p>Built-in authority translation and Access Control List (ACL) mapping. Search engines only return documents that users are explicitly authorized to view.</p>
</div>
<div class="feature-card">
<div class="feature-icon-wrapper">
<svg class="feature-icon" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M13 10V3L4 14h7v7l9-11h-7z"/>
</svg>
</div>
<h3>High Performance & Concurrency</h3>
<p>Leverages Java 25 Virtual Threads and Spring Boot. Processes thousands of streams with minimal memory overhead, preventing source system bottlenecks.</p>
</div>
<div class="feature-card">
<div class="feature-icon-wrapper">
<svg class="feature-icon" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9.663 17h4.673M12 3v1m6.364 1.636l-.707.707M21 12h-1M4 12H3m3.343-5.657l-.707-.707m2.828 9.9a5 5 0 117.072 0l-.548.547A3.374 3.374 0 0014 18.469V19a2 2 0 11-4 0v-.531c0-.895-.356-1.754-.988-2.386l-.548-.547z"/>
</svg>
</div>
<h3>AI & RAG Ready Ingestion</h3>
<p>Advanced RAG support with flexible chunking and embedding logic. Configure multiple embedding strategies (Ollama, OpenAI, HuggingFace) across diverse vector index backends.</p>
</div>
<div class="feature-card">
<div class="feature-icon-wrapper">
<svg class="feature-icon" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M19 11H5m14 0a2 2 0 012 2v6a2 2 0 01-2 2H5a2 2 0 01-2-2v-6a2 2 0 012-2m14 0V9a2 2 0 00-2-2M5 11V9a2 2 0 012-2m0 0V5a2 2 0 012-2h6a2 2 0 012 2v2M7 7h10"/>
</svg>
</div>
<h3>Pluggable SPI Architecture</h3>
<p>Simple interfaces for writing custom connectors. Define a new Repository Source, Transformation filter, or Database Output connector in just a few lines of Java.</p>
</div>
<div class="feature-card">
<div class="feature-icon-wrapper">
<svg class="feature-icon" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 19v-6a2 2 0 00-2-2H5a2 2 0 00-2 2v6a2 2 0 002 2h2a2 2 0 002-2zm0 0V9a2 2 0 012-2h2a2 2 0 012 2v10m-6 0a2 2 0 002 2h2a2 2 0 002-2m0 0V5a2 2 0 012-2h2a2 2 0 012 2v14a2 2 0 002 2h2a2 2 0 002-2z"/>
</svg>
</div>
<h3>AI-Powered Observability (AIOps)</h3>
<p>Integrates OpenTelemetry (OTel) correlated traces and Micrometer metrics. Automatically translates complex pipeline stack traces into human-readable AI Root Cause Analysis (RCA) and diagnostic recommendations. <a href="observability.html" class="learn-more-link">Learn more →</a></p>
</div>
<div class="feature-card">
<div class="feature-icon-wrapper">
<svg class="feature-icon" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M11 5H6a2 2 0 00-2 2v11a2 2 0 002 2h11a2 2 0 002-2v-5m-1.414-9.414a2 2 0 112.828 2.828L11.828 15H9v-2.828l8.586-8.586z"/>
</svg>
</div>
<h3>Auto-Narrativization Copilot</h3>
<p>Configured per job or interactive panel. Translates connector schemas into human-readable <strong>Mustache narrative templates</strong> and mock datasets via Ollama (<code>llama3.2</code>) or OpenAI. Automatically falls back to deterministic generation when AI is offline.</p>
</div>
<div class="feature-card">
<div class="feature-icon-wrapper">
<svg class="feature-icon" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M10 20l4-16m4 4l4 4-4 4M6 16l-4-4 4-4"/>
</svg>
</div>
<h3>Java Client SDK (`oc-java-client-sdk`)</h3>
<p>Strongly typed, fluent <strong>Java 25 & Spring Boot 4</strong> client SDK. Programmatically manage ingestion job lifecycles, connector setups, Copilot templates, and AIOps diagnostics with zero third-party HTTP transport bloat. <a href="sdk.html" class="learn-more-link">Explore SDK →</a></p>
</div>
<div class="feature-card full-width">
<div class="feature-icon-wrapper">
<svg class="feature-icon" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M8 9l3 3-3 3m5 0h3M5 20h14a2 2 0 002-2V6a2 2 0 00-2-2H5a2 2 0 00-2 2v12a2 2 0 002 2z"/>
</svg>
</div>
<h3>Secure Model Context Protocol (MCP) Server</h3>
<p>Exposes security-filtered similarity search and document retrieval tools directly to LLMs and AI agent frameworks using the <strong>Model Context Protocol (MCP)</strong>. Enforces user authentication policies (principals and roles) and ACL checks on the server-side to prevent prompt injection and unauthorized data leakage.</p>
</div>
</div>
</section>
<!-- Interactive Architecture Section -->
<section id="architecture" class="architecture-section">
<div class="section-header">
<h2 class="section-title">Ingestion Architecture</h2>
<p class="section-subtitle">Decoupled, event-driven orchestration with a horizontally scalable <strong>Embedding Microservice</strong> leveraging Kafka and the Claim Check Pattern.</p>
</div>
<div class="architecture-container">
<div class="arch-visual">
<!-- SVG diagram with glowing paths & interactive points -->
<svg id="arch-svg" viewBox="0 0 900 500" fill="none" xmlns="http://www.w3.org/2000/svg">
<defs>
<linearGradient id="primary-grad" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" stop-color="#7C3AED" />
<stop offset="100%" stop-color="#C084FC" />
</linearGradient>
<linearGradient id="secondary-grad" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" stop-color="#06B6D4" />
<stop offset="100%" stop-color="#22D3EE" />
</linearGradient>
<linearGradient id="embed-grad" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" stop-color="#F59E0B" />
<stop offset="100%" stop-color="#FCD34D" />
</linearGradient>
<filter id="glow-effect" x="-20%" y="-20%" width="140%" height="140%">
<feGaussianBlur stdDeviation="6" result="blur" />
<feComposite in="SourceGraphic" in2="blur" operator="over" />
</filter>
</defs>
<!-- Main horizontal flow lines -->
<path class="flow-line line-active" d="M130,230 L220,230" />
<path class="flow-line line-active" d="M370,230 L430,230" />
<path class="flow-line line-active" d="M560,230 L640,230" />
<!-- Kafka to Embedding Service (up) -->
<path class="flow-line line-active" d="M490,190 L490,130" />
<!-- Embedding Service back to Kafka (down) -->
<path class="flow-line line-active" d="M540,130 L540,190" />
<!-- Admin UI to Core (dashed) -->
<path class="flow-line line-dashed" d="M295,185 L295,140" />
<!-- Core to Claim Check Store (down, dashed) -->
<path class="flow-line line-dashed" d="M295,275 L295,320" />
<!-- Vector Writer to MCP (up, dashed) -->
<path class="flow-line line-dashed" d="M810,190 L810,140" />
<!-- DB below Kafka (dashed) -->
<path class="flow-line line-dashed" d="M495,270 L495,320" />
<!-- Security below Vector Writer (dashed) -->
<path class="flow-line line-dashed" d="M760,270 L760,320" />
<!-- Dynamic Ingestion Particles -->
<circle id="anim-particle-1" r="5" fill="#7C3AED" filter="url(#glow-effect)" />
<circle id="anim-particle-2" r="5" fill="#06B6D4" filter="url(#glow-effect)" />
<circle id="anim-particle-3" r="5" fill="#F59E0B" filter="url(#glow-effect)" />
<!-- Node 1: Sources -->
<g class="node-group" data-node="sources">
<rect x="10" y="145" width="120" height="185" rx="10" class="node-bg" />
<text x="70" y="168" class="node-title">Sources</text>
<text x="70" y="188" class="node-sub">File System</text>
<text x="70" y="206" class="node-sub">Apache Iceberg🧊</text>
<text x="70" y="224" class="node-sub">Alfresco ACS</text>
<text x="70" y="242" class="node-sub">Flowable BPMN🔄</text>
<text x="70" y="260" class="node-sub">Camunda BPMN⚡</text>
<text x="70" y="278" class="node-sub">StormCrawler🕷️</text>
</g>
<!-- Node 2: Ingestion Core -->
<g class="node-group" data-node="core">
<rect x="220" y="185" width="150" height="90" rx="10" class="node-bg accent" />
<text x="295" y="212" class="node-title accent-text">Ingestion Core</text>
<text x="295" y="233" class="node-desc">Tika Extractor</text>
<text x="295" y="250" class="node-desc">Token Splitter</text>
<text x="295" y="265" class="node-desc">Publish Chunks</text>
</g>
<!-- Node 3: Apache Kafka -->
<g class="node-group" data-node="kafka">
<rect x="430" y="185" width="130" height="90" rx="10" class="node-bg cyan" />
<text x="495" y="212" class="node-title cyan-text">Apache Kafka</text>
<text x="495" y="233" class="node-desc">Ingestion Topic</text>
<text x="495" y="250" class="node-desc">Chunks Topic</text>
<text x="495" y="265" class="node-desc">Embedded Topic</text>
</g>
<!-- Node 4: Vector Writer / Outputs -->
<g class="node-group" data-node="outputs">
<rect x="640" y="145" width="145" height="145" rx="10" class="node-bg" />
<text x="712" y="168" class="node-title">Vector Writer</text>
<text x="712" y="188" class="node-sub">SeaTunnel Fan-Out 🌊</text>
<text x="712" y="206" class="node-sub">Luxir / Solr 10 ✨</text>
<text x="712" y="224" class="node-sub">Vespa Engine 🚀</text>
<text x="712" y="242" class="node-sub">pgvector / Milvus</text>
<text x="712" y="260" class="node-sub">Qdrant / OpenSearch</text>
</g>
<!-- Node 5: Admin React UI (Top Center-Left) -->
<g class="node-group" data-node="ui">
<rect x="228" y="65" width="134" height="70" rx="8" class="node-bg border-only" />
<text x="295" y="95" class="node-title-small">Admin React UI</text>
<text x="295" y="115" class="node-desc">oc-admin-ui</text>
</g>
<!-- Node 6: oc-embedding-service (Top Center — NEW DECOUPLED SERVICE) -->
<g class="node-group" data-node="embedding">
<rect x="428" y="50" width="164" height="80" rx="8" class="node-bg embed-accent" />
<text x="510" y="75" class="node-title embed-text">oc-embedding-service</text>
<text x="510" y="95" class="node-desc">EmbeddingConsumer</text>
<text x="510" y="111" class="node-desc">EmbeddingModelFactory</text>
</g>
<!-- Node 7: Secure MCP Server (Top Right) -->
<g class="node-group" data-node="mcp">
<rect x="762" y="65" width="128" height="70" rx="8" class="node-bg border-only" />
<text x="826" y="93" class="node-title-small">Secure MCP Server</text>
<text x="826" y="113" class="node-desc">Model Context Protocol</text>
</g>
<!-- Node 8: PostgreSQL / Redis (Bottom Center) -->
<g class="node-group" data-node="db">
<rect x="432" y="320" width="130" height="80" rx="8" class="node-bg border-only" />
<text x="497" y="350" class="node-title-small">PostgreSQL / Redis</text>
<text x="497" y="370" class="node-desc">Job Status Cache</text>
<text x="497" y="385" class="node-desc">Metadata State</text>
</g>
<!-- Node 9: Authority Service (Bottom Right) -->
<g class="node-group" data-node="security">
<rect x="710" y="320" width="120" height="80" rx="8" class="node-bg border-only" />
<text x="770" y="350" class="node-title-small">Authority Service</text>
<text x="770" y="370" class="node-desc">Access Filters</text>
<text x="770" y="385" class="node-desc">ACL Mapping</text>
</g>
<!-- Node 10: Claim Check Store (Bottom Left) -->
<g class="node-group" data-node="claimcheck">
<rect x="225" y="320" width="140" height="80" rx="8" class="node-bg border-only" />
<text x="295" y="348" class="node-title-small">Claim Check Store</text>
<text x="295" y="368" class="node-desc">Ozone Native (ofs RPC)</text>
<text x="295" y="385" class="node-desc">Ozone S3 Gateway (s3g)</text>
</g>
<!-- Scale-out badge on embedding service -->
<rect x="548" y="54" width="38" height="18" rx="9" fill="#F59E0B" opacity="0.9"/>
<text x="567" y="67" text-anchor="middle" fill="#1a1a2e" font-size="9" font-weight="700">×N</text>
</svg>
</div>
<div class="arch-info-card glass" id="arch-detail">
<h3 id="arch-detail-title">System Architecture</h3>
<p id="arch-detail-desc">Hover over or tap any node in the architecture diagram to view how OpenCrawling orchestrates the high-speed data flow and security controls.</p>
</div>
</div>
</section>
<!-- Admin Console Showcase Section -->
<section id="dashboard" class="dashboard-section">
<div class="section-header">
<h2 class="section-title">Unified Administration Console</h2>
<p class="section-subtitle">Manage repository connectors, monitor Kafka topics, and track ingestion jobs from a sleek, real-time dashboard.</p>
</div>
<div class="dashboard-container">
<div class="browser-mockup glass">
<div class="browser-header">
<div class="browser-dots">
<span class="browser-dot red"></span>
<span class="browser-dot yellow"></span>
<span class="browser-dot green"></span>
</div>
<div class="browser-address">
<svg class="lock-icon" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" width="12" height="12"><rect x="3" y="11" width="18" height="11" rx="2" ry="2"/><path d="M7 11V7a5 5 0 0 1 10 0v4"/></svg>
<span>localhost:5173/dashboard</span>
</div>
</div>
<div class="slider-wrapper">
<div class="slides-container" id="screenshot-slider">
<div class="slide active">
<img src="assets/admin-ui-walkthrough.gif" alt="OpenCrawling Admin UI Live Walkthrough Screencast" loading="lazy" />
<div class="slide-caption">
<h3>Interactive Admin UI Screencast Walkthrough</h3>
<p>Experience real-time pipeline management, job scheduling, AIOps diagnostics, and Auto-Narrativization Copilot in action.</p>
</div>
</div>
<div class="slide">
<img src="assets/screenshots/ui-dashboard.png" alt="OpenCrawling Admin Dashboard Telemetry" loading="lazy" />
<div class="slide-caption">
<h3>Real-Time Ingestion Telemetry</h3>
<p>Track crawling health, active workers, Kafka queue load, and document-indexing throughput at a glance.</p>
</div>
</div>
<div class="slide">
<img src="assets/screenshots/ui-pipeline-job-management.png" alt="OpenCrawling Ingestion Job Pipeline" loading="lazy" />
<div class="slide-caption">
<h3>Pipeline & Job Scheduling</h3>
<p>Schedule, execute, pause, and configure crawling operations. Review historical runs, document success rates, and errors.</p>
</div>
</div>
<div class="slide">
<img src="assets/screenshots/ui-connector-configuration.png" alt="OpenCrawling Repository Connector Configuration" loading="lazy" />
<div class="slide-caption">
<h3>Connector Registry Configuration</h3>
<p>Set up SharePoint credentials, S3 bucket access policies, database queries, and custom connector metadata properties.</p>
</div>
</div>
<div class="slide">
<img src="assets/screenshots/ui-ingestion-and-embedding-settings.png" alt="OpenCrawling Embedding Settings" loading="lazy" />
<div class="slide-caption">
<h3>AI & Vector Embedding Mappings</h3>
<p>Select RAG embedding models (local Ollama, OpenAI), set custom token text-splitter boundaries, and map schema attributes.</p>
</div>
</div>
<div class="slide">
<img src="assets/screenshots/ui-real-time-activity-logs.png" alt="OpenCrawling Real-Time Activity Logs" loading="lazy" />
<div class="slide-caption">
<h3>Live System Activity & Audit Logs</h3>
<p>Stream real-time JVM telemetry, Kafka partition consumer offsets, active threads, and document security ACL mapping logs.</p>
</div>
</div>
</div>
<!-- Slider Navigation -->
<button class="slider-arrow prev" id="slider-prev-btn" aria-label="Previous Slide">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round" width="24" height="24"><path d="M15 18l-6-6 6-6"/></svg>
</button>
<button class="slider-arrow next" id="slider-next-btn" aria-label="Next Slide">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round" width="24" height="24"><path d="M9 18l6-6-6-6"/></svg>
</button>
</div>
<div class="slider-dots" id="slider-dots-container">
<span class="slider-dot active" data-index="0"></span>
<span class="slider-dot" data-index="1"></span>
<span class="slider-dot" data-index="2"></span>
<span class="slider-dot" data-index="3"></span>
<span class="slider-dot" data-index="4"></span>
</div>
</div>
</div>
</section>
<!-- Live Ingestion Simulator Section -->
<section id="simulator" class="simulator-section">
<div class="section-header">
<h2 class="section-title">Live Ingestion Simulator</h2>
<p class="section-subtitle">Simulate real-time ingestion, security mapping, and the new <strong>decoupled embedding scale-out</strong> pipeline locally in your browser.</p>
</div>
<div class="simulator-container glass">
<div class="sim-controls">
<div class="sim-group">
<label for="sim-source">1. Select Content Source</label>
<select id="sim-source">
<option value="database">File System</option>
<option value="flowable">Flowable BPMN (Historic Workflow Instances)</option>
<option value="camunda">Camunda BPMN (Historic Workflow Instances)</option>
<option value="alfresco">Alfresco Content Services</option>
<option value="iceberg">Apache Iceberg (Data Lakehouse)</option>
<option value="stormcrawler">Apache StormCrawler (Distributed Web Crawler)</option>
</select>
</div>
<div class="sim-group">
<label for="sim-destination">2. Select Vector Database / Output</label>
<select id="sim-destination">
<option value="seatunnel">Apache SeaTunnel (Distributed Fan-Out)</option>
<option value="luxir">Luxir Hybrid Search Engine</option>
<option value="pgvector">PostgreSQL + pgvector</option>
<option value="milvus">Milvus Standalone</option>
<option value="qdrant">Qdrant Vector Database</option>
<option value="vespa">Vespa Search Engine</option>
<option value="solr">Apache Solr 10 Vector Store</option>
<option value="opensearch">OpenSearch 2.x / 3.x</option>
</select>
</div>
<div class="sim-group">
<label for="sim-embed-scale">3. Embedding Service Replicas</label>
<select id="sim-embed-scale">
<option value="1">1 instance (default)</option>
<option value="2">2 instances (scale ×2)</option>
<option value="3">3 instances (scale ×3)</option>
</select>
</div>
<button id="start-sim-btn" class="btn btn-primary btn-full">Run Ingestion Job</button>
</div>
<div class="sim-display">
<div class="sim-status-bar">
<span class="status-indicator" id="sim-status-dot"></span>
<span id="sim-status-text">Ready to run</span>
</div>
<!-- Live Animation Pipeline Area -->
<div class="sim-pipeline" id="sim-pipeline">
<div class="pipeline-station" id="station-src">
<span class="station-icon">📂</span>
<span class="station-label">Crawler</span>
</div>
<div class="pipeline-connector-line">
<div class="pipeline-pulse" id="pulse-1"></div>
</div>
<div class="pipeline-station" id="station-kafka">
<span class="station-icon">🚇</span>
<span class="station-label">Kafka Broker</span>
</div>
<div class="pipeline-connector-line">
<div class="pipeline-pulse" id="pulse-2"></div>
</div>
<div class="pipeline-station" id="station-tika">
<span class="station-icon">⚙️</span>
<span class="station-label">Ingestion (Tika)</span>
</div>
<div class="pipeline-connector-line">
<div class="pipeline-pulse" id="pulse-3"></div>
</div>
<div class="pipeline-station embed-station" id="station-embed-svc">
<span class="station-icon">🔱</span>
<span class="station-label">oc-embedding-service</span>
<span class="station-badge" id="embed-scale-badge">×1</span>
</div>
<div class="pipeline-connector-line">
<div class="pipeline-pulse" id="pulse-4"></div>
</div>
<div class="pipeline-station" id="station-vector">
<span class="station-icon">🧬</span>
<span class="station-label">Vector Writer</span>
</div>
</div>
<!-- Terminal Logs Area -->
<div class="sim-terminal">
<div class="sim-terminal-header">
<span>Job Console Output</span>
<button id="clear-sim-logs" class="clear-logs-btn">Clear</button>
</div>
<div class="sim-terminal-body" id="sim-logs">
<p class="log-info">Select a source and click "Run Ingestion Job" to start simulation...</p>
</div>
</div>
</div>
</div>
</section>
<!-- Quick Start / Getting Started -->
<section id="getting-started" class="getting-started-section">
<div class="section-header">
<h2 class="section-title">Getting Started in 2 Minutes</h2>
<p class="section-subtitle">Follow these quick steps to get OpenCrawling up and running locally.</p>
</div>
<div class="tabs-container glass">
<div class="tabs-header">
<button class="tab-btn active" data-tab="docker">Docker Compose</button>
<button class="tab-btn" data-tab="maven">Maven Dependency</button>
<button class="tab-btn" data-tab="connector">Custom Connector</button>
</div>
<div class="tabs-content">
<!-- Docker Tab -->
<div class="tab-pane active" id="tab-docker">
<div class="code-wrapper">
<button class="copy-code-btn" aria-label="Copy code block">Copy</button>
<pre><code class="language-bash"># Option 1: Standard Multi-Tier Stack (Backend + Standalone Frontend)
# 1. Start backing services (Postgres, Redis, Kafka, Ollama)
docker compose up -d
# 2. Start pre-built applications from Docker Hub (no compilation)
docker compose -f docker-compose-apps-dockerhub.yml up -d
# 3. (Alternative) Or build from source and run:
docker compose -f docker-compose-apps.yml build
docker compose -f docker-compose-apps.yml up -d
# Option 2: Decoupled Microservices Pipeline (Fully Distributed Workers)
# 1. Run using pre-built Docker Hub images (includes OTel stack)
docker compose -f docker-compose-decoupled-dockerhub.yml up -d
# 2. (Alternative) Or run using pre-built GHCR release images:
docker compose -f docker-compose-decoupled-dist.yml up -d
# 3. (Alternative) Or build from local source and run:
docker compose -f docker-compose-decoupled.yml build
docker compose -f docker-compose-decoupled.yml up -d
# Access points:
# - Standalone Administration Console: http://localhost:3000
# - Secure MCP Server / REST Backend: http://localhost:8080</code></pre>
</div>
</div>
<!-- Maven Tab -->
<div class="tab-pane" id="tab-maven">
<div class="code-wrapper">
<button class="copy-code-btn" aria-label="Copy code block">Copy</button>
<pre><code class="language-xml"><!-- Add core connector API to your spring boot application -->
<dependency>
<groupId>org.opencrawling</groupId>
<artifactId>oc-core</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
<!-- Include FileSystem Repository Connector if required -->
<dependency>
<groupId>org.opencrawling</groupId>
<artifactId>oc-filesystem-repository-connector</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency></code></pre>
</div>
</div>
<!-- Connector Tab -->
<div class="tab-pane" id="tab-connector">
<div class="code-wrapper">
<button class="copy-code-btn" aria-label="Copy code block">Copy</button>
<pre><code class="language-java">// Implement RepositoryConnector interface to create custom sources
public class MyCustomRepositoryConnector implements RepositoryConnector {
@Override
public void connect(Map<String, String> configParams) throws RepositoryConnectionException {
// Establish authentication with target repository
}
@Override
public void scanDocuments(DocumentProcessor processor, JobContext context) {
// Scan documents incrementally and dispatch to processor
for (Document doc : fetchNewDocuments()) {
processor.processDocument(
doc.getId(),
doc.getContentStream(),
doc.getMetadata(),
doc.getAcl() // Synchronize Access Control Lists!
);
}
}
}</code></pre>
</div>
</div>
</div>
</div>
</section>
<!-- Open Ingestion Standard (OIS) Section -->
<section id="standard" class="standard-section">
<div class="section-header">
<h2 class="section-title">Open Ingestion Standard (OIS)</h2>
<p class="section-subtitle">A cross-platform specification for secure enterprise data exchange, document lifecycle operations, and configuration.</p>
</div>
<div class="standard-container glass">
<div class="standard-text-content">
<h3>The Ingestion Manifesto</h3>
<p>Enterprise data pipelines suffer from proprietary lock-in and security permission leaks. The <strong>OIS specification</strong> standardizes document payloads, metadata repositories, workflow processes, and full document lifecycles—now featuring native support for <strong>UPSERT & DELETE tombstones</strong> to automatically purge removed documents from downstream vector stores, alongside <strong>CMIS</strong> and <strong>BPMN</strong> integrations.</p>
<div class="standard-badges">
<span class="std-badge">🔒 Zero-Trust Security</span>
<span class="std-badge">🗑️ Tombstones & Deletion</span>
<span class="std-badge">🗂️ CMIS & BPMN Ready</span>
<span class="std-badge">🔌 Vendor-Neutral</span>
<span class="std-badge">⚙️ Event-Decoupled</span>
</div>
<div class="standard-actions">
<a href="https://github.com/OpenCrawling/open-ingestion-standard" target="_blank" rel="noopener noreferrer" class="btn btn-primary">Read Manifesto</a>
<a href="https://github.com/OpenCrawling/open-ingestion-standard/tree/master/schemas" target="_blank" rel="noopener noreferrer" class="btn btn-secondary">Explore Schemas</a>
</div>
</div>
<div class="standard-visual-content">
<div class="schema-card">
<div class="schema-card-header">
<div style="display: flex; gap: 0.35rem; flex-wrap: wrap;">
<button class="schema-tab-btn active" data-schema="standard">Document</button>
<button class="schema-tab-btn" data-schema="delete">Delete Tombstone</button>
<button class="schema-tab-btn" data-schema="cmis">CMIS</button>
<button class="schema-tab-btn" data-schema="bpmn">BPMN</button>
</div>
<span class="schema-file" id="schema-filename">sample-document.json</span>
</div>
<pre><code class="language-json" id="schema-code-block">{
"id": "doc-554032-sp",
"action": "UPSERT",
"source": { "type": "sharepoint", "instance": "..." },
"content": { "mimeType": "application/pdf", "text": "..." },
"metadata": { "title": "Fiscal Strategy", "author": "..." },
"security": { "inheritanceEnabled": true, "permissions": [] }
}</code></pre>
</div>
</div>
</div>
</section>
<!-- Blog Articles Section -->
<section id="blog" class="blog-section">
<div class="section-header">
<h2 class="section-title">Latest Articles & Deep Dives</h2>
<p class="section-subtitle">Read the latest announcements, engineering insights, and tutorials from the OpenCrawling core team.</p>
</div>
<div class="blog-grid">
<article class="blog-card glass featured-blog-card">
<div class="blog-card-meta">
<span class="blog-badge badge-new" style="background: linear-gradient(135deg, #0284C7, #0369A1); color: #FFF;">Distributed Fan-Out</span>
<span class="blog-date">September 28, 2026</span>
<span class="blog-read-time">6 min read</span>
</div>
<h3 class="blog-card-title"><a href="blog/introducing-apache-seatunnel-output-connector.html">Distributed Fan-Out & Multi-Sink Pipelines: Announcing the Apache SeaTunnel Output Connector</a></h3>
<p class="blog-excerpt">
Discover OpenCrawling's native Apache SeaTunnel integration (<strong>oc-seatunnel-output-connector</strong>). Built on Apache SeaTunnel 2.3.13, automated HOCON DAG synthesis, and Zeta Engine REST API v2 orchestration, delivering parallel streaming fan-out of Open Ingestion Standard (OIS) chunks, vectors, and Zero-Trust ACLs across 100+ destination sinks.
</p>
<div class="blog-card-footer">
<a href="blog/introducing-apache-seatunnel-output-connector.html" class="card-link">Read Article →</a>
</div>
</article>
<article class="blog-card glass">
<div class="blog-card-meta">
<span class="blog-badge" style="background: linear-gradient(135deg, #DC2626, #B91C1C); color: #FFF;">Connector Release</span>
<span class="blog-date">September 28, 2026</span>
<span class="blog-read-time">6 min read</span>
</div>
<h3 class="blog-card-title"><a href="blog/introducing-stormcrawler-repository-connector.html">Enterprise Distributed Web Crawling: Announcing the Apache StormCrawler Repository Connector & Bolt</a></h3>
<p class="blog-excerpt">
Discover OpenCrawling's native Apache StormCrawler integration (<strong>oc-stormcrawler-repository-connector</strong> and <strong>oc-stormcrawler-bolt</strong>). Built on Apache Storm 3.1.0, StormCrawler 3.7.0, and Java 25 Virtual Threads, delivering dual-stream OIS UPSERT and DELETE tombstone dispatching, cryptographic content deduplication, and Nimbus REST cluster supervision.
</p>
<div class="blog-card-footer">
<a href="blog/introducing-stormcrawler-repository-connector.html" class="card-link">Read Article →</a>
</div>
</article>
<article class="blog-card glass">
<div class="blog-card-meta">
<span class="blog-badge" style="background: linear-gradient(135deg, #00ADD8, #0284C7); color: #FFF;">Vector Store</span>
<span class="blog-date">September 22, 2026</span>
<span class="blog-read-time">6 min read</span>
</div>
<h3 class="blog-card-title"><a href="blog/introducing-luxir-output-connector.html">Lightweight Hybrid Search & Dense Vector kNN: Announcing the Luxir Output Connector</a></h3>
<p class="blog-excerpt">
Discover OpenCrawling's native Luxir Output Connector (<strong>oc-luxir-output-connector</strong>). Built with native Java 25 <code>HttpClient</code>, dynamic schema conventions, 1024-dimension dense vector kNN search, Zero-Trust ACL token indexing, and decoupled Kafka writer consumer execution for enterprise RAG.
</p>
<div class="blog-card-footer">
<a href="blog/introducing-luxir-output-connector.html" class="card-link">Read Article →</a>
</div>
</article>
<article class="blog-card glass">
<div class="blog-card-meta">
<span class="blog-badge" style="background: linear-gradient(135deg, #10B981, #059669); color: #FFF;">Migration</span>
<span class="blog-date">September 21, 2026</span>
<span class="blog-read-time">7 min read</span>
</div>
<h3 class="blog-card-title"><a href="blog/introducing-apache-manifoldcf-migrator.html">Migrating Enterprise Crawler Workloads: Announcing the Apache ManifoldCF Migrator</a></h3>
<p class="blog-excerpt">
Discover OpenCrawling's dedicated migration engine and visual Admin UI wizard (<strong>oc-mcf-migrator</strong>). Audit, convert to Open Ingestion Standard (OIS) declarative manifests, and import legacy Apache ManifoldCF crawler configurations into OpenCrawling with full fidelity.
</p>
<div class="blog-card-footer">
<a href="blog/introducing-apache-manifoldcf-migrator.html" class="card-link">Read Article →</a>
</div>
</article>
<article class="blog-card glass">
<div class="blog-card-meta">
<span class="blog-badge" style="background: linear-gradient(135deg, #7C3AED, #9333EA); color: #FFF;">Standard Spec</span>
<span class="blog-date">September 3, 2026</span>
<span class="blog-read-time">5 min read</span>
</div>
<h3 class="blog-card-title"><a href="blog/introducing-ois-tombstone-deletion.html">Standardized Document Lifecycles & Deletion Tombstones: Enhancing the Open Ingestion Standard (OIS)</a></h3>
<p class="blog-excerpt">
Discover how the Open Ingestion Standard (OIS) introduces standardized document lifecycle operations and deletion tombstones (<code>action: DELETE</code>) to ensure downstream vector stores and search engines purge deleted content automatically.
</p>
<div class="blog-card-footer">
<a href="blog/introducing-ois-tombstone-deletion.html" class="card-link">Read Article →</a>
</div>
</article>
<article class="blog-card glass">
<div class="blog-card-meta">
<span class="blog-badge" style="background: linear-gradient(135deg, #D45B07, #FF6600); color: #FFF;">Vector Store</span>
<span class="blog-date">September 3, 2026</span>
<span class="blog-read-time">6 min read</span>
</div>
<h3 class="blog-card-title"><a href="blog/introducing-apache-solr-output-connector.html">Solr 10 Dense Vector Search & HNSW Quantization: Announcing the Apache Solr Output Connector</a></h3>
<p class="blog-excerpt">
Discover OpenCrawling's native Apache Solr Output Connector (<strong>oc-solr-output-connector</strong>). Built with SolrJ 10 native JDK HTTP Client, <code>solr.DenseVectorField</code>, HNSW quantization, <code>efSearch</code> tuning, and REST schema auto-provisioning.
</p>
<div class="blog-card-footer">
<a href="blog/introducing-apache-solr-output-connector.html" class="card-link">Read Article →</a>
</div>
</article>
<article class="blog-card glass">
<div class="blog-card-meta">
<span class="blog-badge" style="background: linear-gradient(135deg, #4285F4, #34A853); color: #FFF;">Transport</span>
<span class="blog-date">August 12, 2026</span>
<span class="blog-read-time">6 min read</span>
</div>
<h3 class="blog-card-title"><a href="blog/introducing-grpc-internal-transport.html">High-Speed Binary Payload Streaming & Zero-Downtime Resilience: Announcing gRPC Internal Transport</a></h3>
<p class="blog-excerpt">
Discover OpenCrawling's new gRPC Internal Transport Layer (<strong>oc-grpc-api</strong>). Featuring HTTP/2 multiplexing, Protobuf binary payload compression up to 50%, AUTO fallback to HTTP/REST, TLS/mTLS encryption, Admin UI hot-reload, and live diagnostic ping probes.
</p>
<div class="blog-card-footer">
<a href="blog/introducing-grpc-internal-transport.html" class="card-link">Read Article →</a>
</div>
</article>
<article class="blog-card glass">
<div class="blog-card-meta">
<span class="blog-badge" style="background: linear-gradient(135deg, #4E9BFA, #3B82F6); color: #FFF;">Vector Store</span>
<span class="blog-date">August 7, 2026</span>
<span class="blog-read-time">6 min read</span>
</div>
<h3 class="blog-card-title"><a href="blog/introducing-vespa-output-connector.html">Hybrid Vector & BM25 Lexical Ranking at Enterprise Scale: Announcing the Vespa Output Connector</a></h3>
<p class="blog-excerpt">
Discover OpenCrawling's native Vespa Output Connector (<strong>oc-vespa-output-connector</strong>). Built with Yahoo's HTTP/2 <code>vespa-feed-client</code>, dynamic multi-dimension vector routing (384/768/1024), ACL pre-filtering, Admin UI Model Insights, and Vespa-native Secure MCP tools.
</p>
<div class="blog-card-footer">
<a href="blog/introducing-vespa-output-connector.html" class="card-link">Read Article →</a>
</div>
</article>
<article class="blog-card glass">
<div class="blog-card-meta">
<span class="blog-badge" style="background: rgba(220,36,76,0.2); color: #FF4D6D;">Vector Store</span>
<span class="blog-date">July 29, 2026</span>
<span class="blog-read-time">5 min read</span>
</div>
<h3 class="blog-card-title"><a href="blog/introducing-qdrant-output-connector.html">Sub-Millisecond Vector Search & Payload ACL Pre-Filtering: Announcing the Qdrant Output Connector</a></h3>
<p class="blog-excerpt">
Discover OpenCrawling's native Qdrant Output Connector (<strong>oc-qdrant-output-connector</strong>). Built with Qdrant's high-performance gRPC Java SDK, scalar quantization, and automatic payload ACL indexing for secure, low-latency enterprise RAG.
</p>
<div class="blog-card-footer">
<a href="blog/introducing-qdrant-output-connector.html" class="card-link">Read Article →</a>
</div>
</article>
<article class="blog-card glass">
<div class="blog-card-meta">
<span class="blog-badge">SDK / Java 25</span>
<span class="blog-date">July 28, 2026</span>
<span class="blog-read-time">5 min read</span>
</div>
<h3 class="blog-card-title"><a href="blog/introducing-java-client-sdk.html">Introducing OpenCrawling Java Client SDK: Programmatic Ingestion Control for Java 25 & Spring Boot</a></h3>
<p class="blog-excerpt">
Discover the official <code>oc-java-client-sdk</code>. Strongly typed Java 25 & Spring Boot 4 client library for programmatically managing document ingestion jobs, connectors, Auto-Narrativization Copilot templates, and AIOps diagnostics.
</p>
<div class="blog-card-footer">
<a href="blog/introducing-java-client-sdk.html" class="card-link">Read Article →</a>
</div>
</article>
<article class="blog-card glass">
<div class="blog-card-meta">
<span class="blog-badge">Maven / Archetypes</span>
<span class="blog-date">July 26, 2026</span>
<span class="blog-read-time">4 min read</span>
</div>
<h3 class="blog-card-title"><a href="blog/introducing-maven-connector-archetypes.html">Introducing Maven Connector Archetypes: Scaffold Custom Connectors in Seconds</a></h3>
<p class="blog-excerpt">
Learn how OpenCrawling's official Maven Archetypes suite allows developers to instantly scaffold ready-to-build Repository, Output, and Transformation Connectors with Docker Compose overlays and Admin UI integration.
</p>
<div class="blog-card-footer">
<a href="blog/introducing-maven-connector-archetypes.html" class="card-link">Read Article →</a>
</div>
</article>
<article class="blog-card glass">
<div class="blog-card-meta">
<span class="blog-badge">Workflow / BPMN</span>
<span class="blog-date">July 24, 2026</span>
<span class="blog-read-time">6 min read</span>
</div>
<h3 class="blog-card-title"><a href="blog/introducing-flowable-camunda-bpmn-connectors.html">Connecting Flowable & Camunda BPMN Engines to Enterprise RAG</a></h3>
<p class="blog-excerpt">
OpenCrawling now natively supports the top open-source BPMN workflow engines: <strong>Flowable</strong> and <strong>Camunda</strong>. Discover how <code>oc-flowable-repository-connector</code> and <code>oc-camunda-repository-connector</code> stream process definitions, historic execution variables, and candidate group ACLs into AI pipelines.
</p>
<div class="blog-card-footer">
<a href="blog/introducing-flowable-camunda-bpmn-connectors.html" class="card-link">Read Article →</a>
</div>
</article>
<article class="blog-card glass">
<div class="blog-card-meta">
<span class="blog-badge">AI / Metadata</span>
<span class="blog-date">July 23, 2026</span>
<span class="blog-read-time">6 min read</span>
</div>
<h3 class="blog-card-title"><a href="blog/auto-narrativization-copilot.html">Auto-Narrativization Copilot: Bridging Structured Schemas to Vector Embeddings</a></h3>
<p class="blog-excerpt">
Learn how OpenCrawling's <strong>Auto-Narrativization Copilot</strong> translates structured database schemas and table records into rich natural language Mustache templates and mock datasets using Spring AI, Ollama (<code>llama3.2</code>), and deterministic offline fallbacks.
</p>
<div class="blog-card-footer">
<a href="blog/auto-narrativization-copilot.html" class="card-link">Read Article →</a>
</div>
</article>
<article class="blog-card glass">
<div class="blog-card-meta">
<span class="blog-badge">Storage</span>
<span class="blog-date">July 21, 2026</span>
<span class="blog-read-time">6 min read</span>
</div>
<h3 class="blog-card-title"><a href="blog/introducing-apache-ozone-claim-check-store.html">Enterprise Object Storage: Announcing Apache Ozone Integration in OpenCrawling</a></h3>
<p class="blog-excerpt">
OpenCrawling now offers full, native support for <strong>Apache Ozone</strong> (v2.2.0+) as a distributed Claim Check store provider. Learn how combining the Claim Check Pattern with Ozone's S3 Gateway eliminates Kafka payload bottlenecks and scales RAG binary ingestion to petabyte scale.