Commit graph

197 commits

Author SHA1 Message Date
Bartosz Sypytkowski
c4514ca5bf chore: add explanation 2024-12-10 10:12:33 +01:00
Bartosz Sypytkowski
9971c23749 fix: update redis cache when inserting collabs in batches 2024-12-10 10:07:54 +01:00
Bartosz Sypytkowski
813179fdb0
chore: Remove async awaits from RealtimeServerActor (#1059)
* chore: reduce number of async await methods in connection manager

* chore: remove async handlers from rt actor
2024-12-10 15:20:28 +08:00
Khor Shu Heng
29cb70a362
fix: synchronization issue due to incorrect collab origin (#1054) 2024-12-09 14:19:00 +08:00
Nathan.fooo
a598aed5ec
chore: remove empty map when query is empty (#1050) 2024-12-07 10:20:53 +08:00
Bartosz Sypytkowski
66deaf3fa3
chore: move collab cache for appflowy collaborate service and add metrics (#1045)
* chore: move collab cache for appflowy collaborate service and add metrics

* chore: add histogram for measuring time spent in postgres transactions

* chore: delete blobs in chunks

* chore: bring back redis cache filling by import worker

* chore: cargo fmt

* chore: make import workspace update existing collabs

* chore: bulk insert collab deduplicate records
2024-12-06 19:55:51 +08:00
Nathan.fooo
ee87bb9bd6
chore: setup tokio console (#1046)
* chore: setup tokio console

* chore: setup tokio console

* chore: clippy

* chore: clippy
2024-12-06 11:33:04 +08:00
Khor Shu Heng
f79042fcc5
Merge pull request #1043 from AppFlowy-IO/web-update-fix
fix: web update should update both disk and mem cache
2024-12-06 10:44:52 +08:00
Nathan.fooo
afcd1130c3
feat: chat settings (#1044)
* feat: chat settings

* chore: fix sqlx
2024-12-05 23:30:11 +08:00
khorshuheng
aa64b7cd6b fix: web update should update both disk and mem cache 2024-12-05 16:51:56 +08:00
Bartosz Sypytkowski
51bf9a85c7
chore: make collab storage switch between s3 and postgres configurable (#1035) 2024-12-03 14:09:09 +08:00
Bartosz Sypytkowski
9ff6f1c744
S3 collab cache (#1028)
* chore: collab cache for S3

* chore: adjust disk cache api to accomodate s3

* chore: move postgres dependent ops to disk cache

* chore: replace blob inserts from pg to s3

* chore: delete blob and collab exist now use s3

* chore: fix clippy erorrs

* chore: post rebase fixes

* chore: fix clippy warnings

* chore: fix imports

* chore: make snapshots work over S3

* chore: remove dead code

* chore: use compressed snapshots

* chore: add zstd compression

* chore: introduce collab size threshold to keep smaller collabs in postgres

* chore: remove collabs from S3 if they were put to postgres

* chore: update tests
2024-12-03 13:08:55 +08:00
Nathan.fooo
3e3bf693fd
chore: upload api endpoint for web upload (#1024)
* chore: upload api endpoint for web upload

* chore: use rust 1.78

* fix: test
2024-11-28 09:09:09 +08:00
Bartosz Sypytkowski
4fb6df5c14
chore: reduce number of SQL statements executed when inserting/updating collab (#1022)
* chore: reduce number of SQL statements executed when inserting new collab

* chore: add workspace id check

* chore: fix previous query

* chore: upsert collabs and embddings as stored procedures

* chore: fix clippy errors

* chore: fix stored procedure to compare workspaces correctly
2024-11-27 13:00:00 +08:00
Nathan.fooo
0ec12c5f2f
chore: add file size (#1015) 2024-11-22 16:31:50 +08:00
Nathan.fooo
1e18180e9d
chore: set max import zip file size (#1011)
* chore: set max import zip file size

* chore: fix test
2024-11-20 14:07:36 +08:00
Nathan.fooo
fd98351473
chore: adjust redis env (#1003) 2024-11-17 14:26:59 +08:00
Bartosz Sypytkowski
3799966f12
chore: store pending collab writes in memory (#1000)
* chore: write immediatelly actually writes immediatelly

* chore: fix clippy errors

* chore: add metrics to new storage queue impl

* chore: set collab batch write capacity to the same as on main branch
2024-11-17 13:25:42 +08:00
Nathan.fooo
d798c81ba4
chore: support split by text len (#1002)
* chore: support split by text len

* chore: update docs

* chore: update tests
2024-11-17 12:45:20 +08:00
Nathan.fooo
dcbc84dacc
chore: recreate group if it isn't exist (#1001) 2024-11-17 12:14:50 +08:00
Nathan.fooo
655f13bc27
chore: search and chat (#999)
* chore: add test for search and chat

* chore: update test

* chore: update test

* chore: update ci

* chore: fix security audio

* chore: multiple core docker build

* chore: multiple core docker build

* chore: update ci

* chore: update model setting

* chore: test ci

* chore: use tiktoken to calcualte token length

* chore: remove env

* chore: use spawn_blocking with condition

* chore: docs

* chore: clippy

* chore: clippy

* chore: docker logs

* chore: pass message id

* chore: clippy
2024-11-16 14:52:12 +08:00
Khor Shu Heng
d0c212ff10
refactor: remove enforcer group (#948) 2024-11-15 14:08:12 +08:00
nathan
b10a0e7829 chore: fix typo 2024-11-15 13:45:25 +08:00
Nathan.fooo
97f9ff3dd8
fix: byte index 8000 is not a char boundary (#995)
* chore: fix split text boundary error and add related tests

* chore: reduce clone
2024-11-15 13:38:08 +08:00
Zack
d8075a9368
Merge pull request #994 from AppFlowy-IO/fix/smtp-email-username
feat: separate smtp username from email
2024-11-14 22:00:43 +08:00
Zack Fu Zi Xiang
4b2eb1b1ba
feat: separate smtp username from email 2024-11-14 16:00:15 +08:00
Nathan.fooo
06a6ea2985
chore: retry upload file when get s3 service error (#993)
* chore: retry upload file when get s3 service error

* chore: ingore upload payload error
2024-11-14 14:39:36 +08:00
Bartosz Sypytkowski
f6fef9918b
chore: split document content into multiple embeddings (#990)
* chore: split document content into multiple embeddings

* chore: fallback to uuid v4 generation due to unmaintained transient dependency

* chore: fix lint errors
2024-11-14 14:15:52 +08:00
Nathan.fooo
50d519c4ea
fix: upload file zip file expire (#987) 2024-11-13 14:57:27 +08:00
Nathan.fooo
91b0c50722
chore: remove logs (#986) 2024-11-13 13:59:23 +08:00
Nathan.fooo
9778843746
chore: disable history service (#985)
* chore: disable history service

* chore: fix build
2024-11-13 10:46:15 +08:00
Nathan.fooo
14adf250a8
chore: update ai api endpoint & rename structs (#953)
* chore: revmap loader type

* chore: revamp context

* chore: fix test

* chore: remove unused handler

* chore: create new chat api endpoint

* chore: add docs

* chore: clippy

* chore: fix test
2024-11-11 20:10:46 +08:00
Bartosz Sypytkowski
6aed9609b9 chore: removed realtime shared state tests 2024-11-04 09:21:10 +01:00
Bartosz Sypytkowski
356cf5d132 chore: fix clippy errors 2024-11-04 08:29:32 +01:00
Bartosz Sypytkowski
f105c2dbc2 chore: fix clippy errors 2024-11-04 08:20:56 +01:00
Bartosz Sypytkowski
f9397eaaf2 chore: remove realtime shared state 2024-11-04 07:58:16 +01:00
Nathan.fooo
a5a5a6b59f
fix: import notion file base cases (#957) 2024-11-02 16:50:51 +08:00
Nathan.fooo
9629d4cefa
chore: add limit for importing zip file (#938)
* chore: add limitation for import zip file

* chore: support upload big file

* chore: implement client api

* chore: implement client api

* chore: implement client api

* chore: update logs

* chore: check file size

* chore: last process at

* chore: set content type

* chore: fix test

* chore: try test

* chore: temporary disable test
2024-10-28 08:51:34 +08:00
Nathan.fooo
246cbd3ec4
chore: update nginx config for import (#935) 2024-10-25 16:47:29 +08:00
Nathan.fooo
e738508d79
fix: Support notion database row document (#929)
* chore: fix import notion database
2024-10-24 19:04:06 +08:00
khorshuheng
16fb9584f9 fix: masked postgres password in logs 2024-10-24 00:34:31 +08:00
Khor Shu Heng
9eb1c36c92
chore: access control interface to return error instead of boolean (#914) 2024-10-21 20:20:14 +08:00
Nathan.fooo
57c44818e2
feat: Import metrics (#911)
* chore: implement metrics for import

* chore: add metrics
2024-10-20 11:05:22 +08:00
Nathan.fooo
b1f37dbbf6
chore: fix petential unzip issues (#910) 2024-10-20 10:48:09 +08:00
Nathan.fooo
773f17e524
fix: import zip without root dir (#898)
* fix: import notion zip file that do not have root dir

* fix: compile
2024-10-18 00:51:53 +08:00
Nathan.fooo
ffee47d0f7
chore: remove workspace and collab listener (#893) 2024-10-17 17:43:00 +08:00
Nathan.fooo
3623d9f296
chore: check file md5 before import (#895) 2024-10-17 17:18:07 +08:00
Khor Shu Heng
8ab5da4aef
fix: limit concurrency level of batch get encode collab (#892) 2024-10-17 13:29:27 +08:00
Nathan.fooo
9942e68089
chore: imported workspace should not become the latest visting workspace (#890)
* chore: imported workspace should not become the latest visting workspace

* chore: sqlx files

* chore: update test

* chore: fmt
2024-10-16 21:02:05 +08:00
Nathan.fooo
d3e4a6870d
chore: template (#889)
* chore: template

* fix: auto del file lifetime

* chore: delete file if fail to write to disk

* fix: modified html

* chore: set content type

* chore: md5 content

* chore: clippy

* fix: image loaded failed

* fix: build

* chore: bump collab

---------

Co-authored-by: Kilu <lu@appflowy.io>
2024-10-16 17:39:24 +08:00