Sharepoint (#2283)

* feat: add Microsoft Entra ID integration

- Updated .env-template and settings.py for Microsoft Entra ID configuration.
- Enhanced ConnectorsCallback to support SharePoint authentication.
- Introduced SharePointAuth and SharePointLoader classes.
- Added required dependencies in requirements.txt.

* feat: agent templates and seeding premade agents (#1910)

* feat: agent templates and seeding premade agents

* fix: ensure ObjectId is used for source reference in agent configuration

* fix: improve source handling in DatabaseSeeder and update tool config processing

* feat: add prompt handling in DatabaseSeeder for agent configuration

* Docs premade agents

* link to prescraped docs

* feat: add template agent retrieval and adopt agent functionality

* feat: simplify agent descriptions in premade_agents.yaml  added docs

---------

Co-authored-by: Pavel <pabin@yandex.ru>
Co-authored-by: Alex <a@tushynski.me>

* feat: add GitHub access token support and fix file content fetching logic (#2032)

* feat: add init for Share Point connector module

* chore(deps): bump mermaid from 11.6.0 to 11.12.0 in /frontend

Bumps [mermaid](https://github.com/mermaid-js/mermaid) from 11.6.0 to 11.12.0.
- [Release notes](https://github.com/mermaid-js/mermaid/releases)
- [Commits](https://github.com/mermaid-js/mermaid/compare/mermaid@11.6.0...mermaid@11.12.0)

---
updated-dependencies:
- dependency-name: mermaid
  dependency-version: 11.12.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Feat: Notification section (#2033)

* Feature/Notification-section

* fix notification ui and add local storage variable to save the state

* add notification component to app.tsx

* refactor: remove MICROSOFT_REDIRECT_URI and update SharePointAuth to use CONNECTOR_REDIRECT_BASE_URI

* feat: Add button to cancel LLM response (#1978)

* feat: Add button to cancel LLM response
- Replace text area with cancel button when loading.
- Add useEffect to change elipsis in cancel button text.
- Add new SVG icon for cancel response.
- Button colors match Figma designs.

* fix: Cancel button UI matches new design
- Delete cancel-response svg.
- Change previous cancel button to match the new Figma design.
- Remove console log in handleCancel function.

* fix: Adjust cancel button rounding

* feat: Update UI for send button
- Add SendArrowIcon component, enables dynamic svg color changes
- Replace original icon
- Update colors and hover effects

* (fix:send-button) minor blink in transition

---------

Co-authored-by: Manish Madan <manishmadan321@gmail.com>

* feat: add SharePoint integration with session validation and UI components

* (feat:oneDrive) file loading for ingestion

* feat(oneDrive): shared user files

* (feat:oneDrive) rm shared file support, as sharedWithMe is degraded

* (feat:sharepoint) shared files for work msa

* (feat:sharepoint) retry on auth failure, decorator

* (fix) tests/ruff

* test: fix sharepoint loader expecting client id

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Abhishek Malviya <abfeb8@gmail.com>
Co-authored-by: Siddhant Rai <47355538+siiddhantt@users.noreply.github.com>
Co-authored-by: Pavel <pabin@yandex.ru>
Co-authored-by: Alex <a@tushynski.me>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Mariam Saeed <69825646+Mariam-Saeed@users.noreply.github.com>
Co-authored-by: Rahul <rahulgithub96@gmail.com>

tests/parser/remote/test_share_point_loader.py

@@ -0,0 +1,200 @@
+"""Tests for SharePoint loader."""
+
+from unittest.mock import patch, MagicMock
+
+from application.parser.connectors.share_point.loader import SharePointLoader
+
+
+def make_response(json_data=None, status_code=200, raise_error=None):
+    resp = MagicMock()
+    resp.status_code = status_code
+    resp.json.return_value = json_data
+    resp.content = b"test content"
+    if raise_error is not None:
+        resp.raise_for_status.side_effect = raise_error
+    else:
+        resp.raise_for_status.return_value = None
+    return resp
+
+
+class TestSharePointLoaderProcessFile:
+    """Test _process_file method."""
+
+    def test_size_retrieved_from_root_level(self):
+        """Should retrieve size from root of file_metadata, not nested file object."""
+        loader = SharePointLoader.__new__(SharePointLoader)
+
+        file_metadata = {
+            "id": "test-id",
+            "name": "test.txt",
+            "createdDateTime": "2024-01-01T00:00:00Z",
+            "lastModifiedDateTime": "2024-01-01T00:00:00Z",
+            "size": 1024,
+            "file": {
+                "mimeType": "text/plain"
+            }
+        }
+
+        doc = loader._process_file(file_metadata, load_content=False)
+
+        assert doc is not None
+        assert doc.extra_info["size"] == 1024
+        assert doc.extra_info["file_name"] == "test.txt"
+        assert doc.extra_info["mime_type"] == "text/plain"
+
+    def test_size_null_when_missing(self):
+        """Should return None when size field is missing."""
+        loader = SharePointLoader.__new__(SharePointLoader)
+
+        file_metadata = {
+            "id": "test-id",
+            "name": "test.txt",
+            "createdDateTime": "2024-01-01T00:00:00Z",
+            "lastModifiedDateTime": "2024-01-01T00:00:00Z",
+            "file": {
+                "mimeType": "text/plain"
+            }
+        }
+
+        doc = loader._process_file(file_metadata, load_content=False)
+
+        assert doc is not None
+        assert doc.extra_info["size"] is None
+
+
+class TestSharePointLoaderLoadFileById:
+    """Test _load_file_by_id method."""
+
+    @patch("application.parser.connectors.share_point.loader.requests.get")
+    @patch("application.parser.connectors.share_point.loader.SharePointAuth.get_token_info_from_session")
+    @patch("application.parser.connectors.share_point.loader.SharePointAuth.__init__", return_value=None)
+    @patch("application.parser.connectors.share_point.loader.SharePointLoader._ensure_valid_token")
+    def test_load_file_by_id_includes_size_in_select(self, mock_ensure_token, mock_auth_init, mock_get_token, mock_get):
+        """Should include size field in $select parameter."""
+        mock_get_token.return_value = {
+            "access_token": "test-token",
+            "refresh_token": "test-refresh"
+        }
+        mock_get.return_value = make_response({
+            "id": "test-id",
+            "name": "test.txt",
+            "createdDateTime": "2024-01-01T00:00:00Z",
+            "lastModifiedDateTime": "2024-01-01T00:00:00Z",
+            "size": 2048,
+            "file": {
+                "mimeType": "text/plain"
+            }
+        })
+
+        loader = SharePointLoader("test-session")
+        doc = loader._load_file_by_id("test-id", load_content=False)
+
+        assert doc is not None
+        assert doc.extra_info["size"] == 2048
+
+        call_args = mock_get.call_args
+        params = call_args[1]["params"]
+        assert "size" in params["$select"]
+
+    @patch("application.parser.connectors.share_point.loader.requests.get")
+    @patch("application.parser.connectors.share_point.loader.SharePointAuth.get_token_info_from_session")
+    @patch("application.parser.connectors.share_point.loader.SharePointAuth.__init__", return_value=None)
+    @patch("application.parser.connectors.share_point.loader.SharePointLoader._ensure_valid_token")
+    def test_load_file_by_id_returns_document_with_size(self, mock_ensure_token, mock_auth_init, mock_get_token, mock_get):
+        """Should return document with size from API response."""
+        mock_get_token.return_value = {
+            "access_token": "test-token",
+            "refresh_token": "test-refresh"
+        }
+        mock_get.return_value = make_response({
+            "id": "test-id",
+            "name": "document.pdf",
+            "createdDateTime": "2024-01-01T00:00:00Z",
+            "lastModifiedDateTime": "2024-06-15T10:30:00Z",
+            "size": 56789,
+            "file": {
+                "mimeType": "application/pdf"
+            }
+        })
+
+        loader = SharePointLoader("test-session")
+        doc = loader._load_file_by_id("test-id", load_content=False)
+
+        assert doc is not None
+        assert doc.doc_id == "test-id"
+        assert doc.extra_info["file_name"] == "document.pdf"
+        assert doc.extra_info["mime_type"] == "application/pdf"
+        assert doc.extra_info["size"] == 56789
+        assert doc.extra_info["created_time"] == "2024-01-01T00:00:00Z"
+        assert doc.extra_info["modified_time"] == "2024-06-15T10:30:00Z"
+        assert doc.extra_info["source"] == "share_point"
+
+
+class TestSharePointLoaderListItems:
+    """Test _list_items_in_parent method."""
+
+    @patch("application.parser.connectors.share_point.loader.requests.get")
+    @patch("application.parser.connectors.share_point.loader.SharePointAuth.get_token_info_from_session")
+    @patch("application.parser.connectors.share_point.loader.SharePointAuth.__init__", return_value=None)
+    @patch("application.parser.connectors.share_point.loader.SharePointLoader._ensure_valid_token")
+    def test_list_items_includes_size_in_select(self, mock_ensure_token, mock_auth_init, mock_get_token, mock_get):
+        """Should include size field in $select parameter when listing items."""
+        mock_get_token.return_value = {
+            "access_token": "test-token",
+            "refresh_token": "test-refresh"
+        }
+        mock_get.return_value = make_response({
+            "value": [
+                {
+                    "id": "file-1",
+                    "name": "file1.txt",
+                    "createdDateTime": "2024-01-01T00:00:00Z",
+                    "lastModifiedDateTime": "2024-01-01T00:00:00Z",
+                    "size": 12345,
+                    "file": {
+                        "mimeType": "text/plain"
+                    }
+                }
+            ]
+        })
+
+        loader = SharePointLoader("test-session")
+        docs = loader._list_items_in_parent("parent-id", limit=10, load_content=False)
+
+        assert len(docs) == 1
+        assert docs[0].extra_info["size"] == 12345
+
+        call_args = mock_get.call_args
+        params = call_args[1]["params"]
+        assert "size" in params["$select"]
+
+    @patch("application.parser.connectors.share_point.loader.requests.get")
+    @patch("application.parser.connectors.share_point.loader.SharePointAuth.get_token_info_from_session")
+    @patch("application.parser.connectors.share_point.loader.SharePointAuth.__init__", return_value=None)
+    @patch("application.parser.connectors.share_point.loader.SharePointLoader._ensure_valid_token")
+    def test_list_items_folders_include_size(self, mock_ensure_token, mock_auth_init, mock_get_token, mock_get):
+        """Should include size for folders as well."""
+        mock_get_token.return_value = {
+            "access_token": "test-token",
+            "refresh_token": "test-refresh"
+        }
+        mock_get.return_value = make_response({
+            "value": [
+                {
+                    "id": "folder-1",
+                    "name": "MyFolder",
+                    "createdDateTime": "2024-01-01T00:00:00Z",
+                    "lastModifiedDateTime": "2024-01-01T00:00:00Z",
+                    "size": 0,
+                    "folder": {}
+                }
+            ]
+        })
+
+        loader = SharePointLoader("test-session")
+        docs = loader._list_items_in_parent("parent-id", limit=10, load_content=False)
+
+        assert len(docs) == 1
+        assert docs[0].extra_info["is_folder"] is True
+        assert docs[0].extra_info["size"] == 0
+