Commit graph

172 commits

Author SHA1 Message Date
Nathan.fooo
ef40f8faa4
chore: adjust chunk size (#1334)
* chore: adjust chunk size

* chore: deduplcate search results

* chore: fix test

* chore: update logs

* chore: separate search and summary

* chore: fix test
2025-04-12 22:22:17 +08:00
Nathan.fooo
c76d2f0b4b
feat: summary search result (#1326)
* chore: summary search result

* chore: add test

* chore: clippy

* chore: distance and score

* chore: update test and clarify score and distance

* chore: fix test

* chore: model env
2025-04-11 20:30:11 +08:00
Bartosz Sypytkowski
1fd900d994
Document paragraphs (#1119)
* chore: create embeddings by paragraphs

* chore: use document paragraphs method

* chore: document indexing by paragraphs with consistent hash

* chore: compare produced embeddings against existing ones

* chore: make pg stored proc compare between input and existing embedded fragments

* chore: missing sqlx generation

* fix: appflowy worker

* chore: make sure that embeddings are only changed when content had changed

* chore: remove partition key and recreate af_collab_embeddings_upsert migration

* chore: use pg15 on CI and update af_collab_embeddings table primary key

* chore: fix test

---------

Co-authored-by: Nathan <nathan@appflowy.io>
2025-04-06 17:47:02 +08:00
Nathan.fooo
7a577492da
chore: parser response (#1319)
* chore: parser response

* chore: clippy
2025-04-05 13:06:37 +08:00
Bartosz Sypytkowski
95bd71cf90 chore: cargo clippy fix 2025-04-01 08:22:36 +02:00
Bartosz Sypytkowski
7b0c232242 chore: cargo clippy fix 2025-04-01 08:22:36 +02:00
Bartosz Sypytkowski
920c4836d7 chore: cargo fmt 2025-04-01 08:22:36 +02:00
Bartosz Sypytkowski
1db7cb36e0 chore: cargo clippy fix 2025-04-01 08:22:36 +02:00
Bartosz Sypytkowski
477d73ffa3 chore: cargo clippy fix 2025-04-01 08:22:36 +02:00
Bartosz Sypytkowski
6d2e8b7164 chore: cargo clippy fix 2025-04-01 08:22:34 +02:00
Bartosz Sypytkowski
d02b1123e1 chore: fixed test compilation errors 2025-04-01 08:19:42 +02:00
Bartosz Sypytkowski
f6060cb82d chore: moving string to uuid in tests 2025-04-01 08:18:12 +02:00
khorshuheng
9c435d9016 feat: add favorite reorder API 2025-03-24 12:55:22 +08:00
khorshuheng
fb15b79909 chore: remove unused collab member crud api 2025-02-13 10:48:19 +08:00
Khor Shu Heng
832bd7f9c9
fix: children of trash views and other user's private space should not appear in search (#1146) 2025-01-10 16:30:35 +08:00
Khor Shu Heng
fa9d53461b
feat: add web guide as part of getting started guide (#1142) 2025-01-10 16:30:05 +08:00
Nathan.fooo
49e83c6b6b
chore: format chat response (#1125)
* chore: chat response with format

* chore: update prompt

* chore: update test

* chore: update test

* chore: fix stress test

* chore: fix test

* chore: test

* chore: test

* chore: fix stress test

* chore: fix test
2025-01-06 01:19:00 +08:00
nathan
1a5c1714db Merge branch 'main' into stateless 2024-12-24 14:59:04 +08:00
Nathan.fooo
1131818eb7
fix: support index document in the background, fix stack overflow when calling rayon::spawn (#1099)
* chore: batch index

* chore: format log

* chore: index workspace

* chore: fix stack overflow

* chore: background index

* chore: clippy

* chore: filter tasks

* chore: clippy

* chore: add metrics

* chore: fix test
2024-12-24 14:30:17 +08:00
Bartosz Sypytkowski
35430c6be2 chore: remove doing snapshots in collab group periodic saves 2024-12-23 13:02:23 +01:00
Bartosz Sypytkowski
9d6b272656 chore: merge main into stateless branch 2024-12-19 06:52:20 +01:00
Zack
c0f7b1d3a9
feat: upsert database row (#1067)
* feat: upsert database row

* feat: improve api to accept database row document payload

* feat: test case for upsert

* chore: refactor

* feat: database row doc content impl

* refactor: simplify insert row field names

* feat: folder updates for database row document

* refactor: simplify row detail return

* feat: add doc contents for database row detail

* feat: refactor database row creation

* feat: upsert database row doc modification

* feat: ignore empty string

* feat: add document similarity check

* chore: refactor to use broadcast with timeout

* feat: wrap broadcast timeout in a tokio spawn
2024-12-18 13:47:20 +08:00
Bartosz Sypytkowski
24deac35a1 Merge branch 'main' into stateless 2024-12-17 06:48:56 +01:00
Nathan.fooo
91c2a925bc
test: add test (#1077) 2024-12-16 16:19:41 +08:00
Bartosz Sypytkowski
1b35326f0a Merge branch 'main' into stateless 2024-12-16 04:47:26 +01:00
Nathan.fooo
ec9c38254b
chore: open ai call directly (#1074)
* chore: use open ai embedding

* chore: update test

* chore: retry

* chore: metrics

* chore: clippy

* chore: env

* chore: delete test
2024-12-15 23:29:46 +08:00
Nathan.fooo
2ab1e94b99
chore: create embed in background (#1071)
* refactor: create indexer scheduler

* refactor: replace current calls

* chore: use rayon thread pool

* chore: index unindexed collab using thread pool

* chore: index collab when write to disk

* chore: add test

* chore: add test

* chore: add test

* chore: fix client api deps

* chore: chat with document test

* chore: fix test

* chore: clippy

* chore: update test
2024-12-15 12:29:51 +08:00
khorshuheng
c1c5bf7420 chore: stop writing to af_collab_member 2024-12-13 10:17:22 +08:00
Bartosz Sypytkowski
41e9302ff9 Merge branch 'main' into stateless 2024-12-12 16:00:45 +01:00
Nathan.fooo
af38efe6d9
feat: sync document through http request (#1064)
* chore: query embedding

* chore: create embeddings

* chore: apply update to editing collab

* refactor: web-update

* chore: calculate missing update the sv is not none

* chore: add test

* chore: fix audit

* chore: commit sqlx

* chore: fix client api

* test: add

* chore: clippy

* chore: fix collab drop when save
2024-12-12 14:53:07 +08:00
Zack
254bc23a6e
feat: insert database row into selected database (#1029)
* feat: insert database row into selected database

* feat: inserting database row

* feat: add impl for cell

* feat: insert row selection and mulitselection

* feat: support datetime field type

* fix: other fields like rich text

* feat: add database field server impl

* feat: add client api and tests

* feat: use to json value impl from collab

* feat: use add database cell impl from collab

* feat: update to latest collab

* chore: upgrade collab and fix tests

* chore: review issues

* fix: code review feedback

* feat: filter only allow supported field types

* feat: support more field types

* feat: support created at and last modified

* feat: add timestamp cell for created at and modified at fields

* chore: linting and formatting

* fix: add created at and last modified

* fix: ci add service dependency of appflowy cloud on admin frontend

* chore: trigger ci

* feat: add logging for admin frontend client signin

* fix: server logs if error

* fix: create admin confirmation without email

* fix: ci add service dependency of appflowy cloud on admin frontend

* fix: server logs if error

* chore: update collab
2024-12-11 15:47:29 +08:00
Nathan.fooo
14b2f3c985
chore: enable running stress on on self-hosted runner (#1053)
* chore: run stress test on selfhost runner

* chore: Update stress_test.yml

* chore: update env

* chore: skip stress_test when running integration test
2024-12-11 09:43:47 +08:00
Bartosz Sypytkowski
66deaf3fa3
chore: move collab cache for appflowy collaborate service and add metrics (#1045)
* chore: move collab cache for appflowy collaborate service and add metrics

* chore: add histogram for measuring time spent in postgres transactions

* chore: delete blobs in chunks

* chore: bring back redis cache filling by import worker

* chore: cargo fmt

* chore: make import workspace update existing collabs

* chore: bulk insert collab deduplicate records
2024-12-06 19:55:51 +08:00
Bartosz Sypytkowski
5e7b5f87fa chore: merge branch main into stateless 2024-12-03 09:10:01 +01:00
Bartosz Sypytkowski
9ff6f1c744
S3 collab cache (#1028)
* chore: collab cache for S3

* chore: adjust disk cache api to accomodate s3

* chore: move postgres dependent ops to disk cache

* chore: replace blob inserts from pg to s3

* chore: delete blob and collab exist now use s3

* chore: fix clippy erorrs

* chore: post rebase fixes

* chore: fix clippy warnings

* chore: fix imports

* chore: make snapshots work over S3

* chore: remove dead code

* chore: use compressed snapshots

* chore: add zstd compression

* chore: introduce collab size threshold to keep smaller collabs in postgres

* chore: remove collabs from S3 if they were put to postgres

* chore: update tests
2024-12-03 13:08:55 +08:00
Bartosz Sypytkowski
d0f9059f3f Merge branch 'main' into stateless 2024-11-20 08:36:28 +01:00
Khor Shu Heng
afeaeb7796
chore: simplify collab level access control (#1008) 2024-11-20 12:29:16 +08:00
Bartosz Sypytkowski
3799966f12
chore: store pending collab writes in memory (#1000)
* chore: write immediatelly actually writes immediatelly

* chore: fix clippy errors

* chore: add metrics to new storage queue impl

* chore: set collab batch write capacity to the same as on main branch
2024-11-17 13:25:42 +08:00
Nathan.fooo
655f13bc27
chore: search and chat (#999)
* chore: add test for search and chat

* chore: update test

* chore: update test

* chore: update ci

* chore: fix security audio

* chore: multiple core docker build

* chore: multiple core docker build

* chore: update ci

* chore: update model setting

* chore: test ci

* chore: use tiktoken to calcualte token length

* chore: remove env

* chore: use spawn_blocking with condition

* chore: docs

* chore: clippy

* chore: clippy

* chore: docker logs

* chore: pass message id

* chore: clippy
2024-11-16 14:52:12 +08:00
Bartosz Sypytkowski
b9ca480143 chore: add retry limit to which collab snapshot can be made 2024-11-07 13:12:17 +01:00
Bartosz Sypytkowski
bc49d73b40 chore: split redis stream readers to active and one shot 2024-11-07 11:43:19 +01:00
Bartosz Sypytkowski
f3cd597832 chore: fix clippy errors 2024-11-06 10:57:32 +01:00
Bartosz Sypytkowski
cc149472de chore: modify stress tests for continuous integration environment 2024-11-05 07:48:14 +01:00
Bartosz Sypytkowski
99203cc7f3 chore: fix flaky tests 2024-11-05 07:48:14 +01:00
Bartosz Sypytkowski
3ad2c3dacb chore: clippy warnings 2024-11-05 07:48:14 +01:00
Bartosz Sypytkowski
832e140f61 chore: collab stress test - make number of edits configurable 2024-11-05 07:48:14 +01:00
Bartosz Sypytkowski
7dc90904cb chore: fix clippy warnings 2024-11-05 07:47:17 +01:00
Bartosz Sypytkowski
f684d3623d chore: fixed collab ack 2024-11-05 07:47:17 +01:00
Bartosz Sypytkowski
fc1a7d28e2 chore: better stress test scenario 2024-11-05 07:47:17 +01:00
Bartosz Sypytkowski
a2d3d7feb5 chore: add perf tests 2024-11-05 07:47:17 +01:00