Berkeley DB Reference Guide: Retrieving records in bulk

Berkeley DB Reference Guide:
Access Methods

Retrieving records in bulk

When retrieving large numbers of records from the database, the number of method calls can often dominate performance. Berkeley DB offers bulk get interfaces which can significantly increase performance for some applications. To retrieve records in bulk, an application buffer must be specified to the DB->get or DBcursor->c_get methods. This is done in the C API by setting the data and ulen fields of the data DBT to reference an application buffer, and the flags field of that structure to DB_DBT_USERMEM. In the Berkeley DB C++ and Java APIs, the actions are similar, although there are API-specific methods to set the DBT values. Then, the DB_MULTIPLE or DB_MULTIPLE_KEY flags are specified to the DB->get or DBcursor->c_get methods, which cause multiple records to be returned in the specified buffer.

(디비로부터 많은 양의 레코드를 읽을때의 많은 메소드함수는 성능을 하락시킨다.버클리디비는 벌크로 읽기 인터페이스를 제공하여 이런문제의 성능을 향상시킨다.레코드를 벌크로 읽기위해서 애플리케이션 버퍼가 DB->get or DBcursor->c_get에 설정되어야 한다.이것은 data DBT의 data and ulen 필드를 애플리케이션 버퍼로 설정하는것이다.그리고 DBT의 flags 필드는 DB_DBT_USERMEM로 설정한다.)

The difference between DB_MULTIPLE and DB_MULTIPLE_KEY is as follows: DB_MULTIPLE returns multiple data items for a single key. For example, the DB_MULTIPLE flag would be used to retrieve all of the duplicate data items for a single key in a single call. The DB_MULTIPLE_KEY flag is used to retrieve multiple key/data pairs, where each returned key may or may not have duplicate data items.

Once the DB->get or DBcursor->c_get method has returned, the application will walk through the buffer handling the returned records. This is implemented for the C and C++ APIs using four macros: DB_MULTIPLE_INIT, DB_MULTIPLE_NEXT, DB_MULTIPLE_KEY_NEXT, and DB_MULTIPLE_RECNO_NEXT. For the Java API, this is implemented as three iterator classes: DbMultipleDataIterator , DbMultipleKeyDataIterator , and DbMultipleRecnoDataIterator .

The DB_MULTIPLE_INIT macro is always called first. It initializes a local application variable and the data DBT for stepping through the set of returned records. Then, the application calls one of the remaining three macros: DB_MULTIPLE_NEXT, DB_MULTIPLE_KEY_NEXT, and DB_MULTIPLE_RECNO_NEXT.

If the DB_MULTIPLE flag was specified to the DB->get or DBcursor->c_get method, the application will always call the DB_MULTIPLE_NEXT macro. If the DB_MULTIPLE_KEY flag was specified to the DB->get or DBcursor->c_get method, and, the underlying database is a Btree or Hash database, the application will always call the DB_MULTIPLE_KEY_NEXT macro. If the DB_MULTIPLE_KEY flag was specified to the DB->get or DBcursor->c_get method, and, the underlying database is a Queue or Recno database, the application will always call the DB_MULTIPLE_RECNO_NEXT macro. The DB_MULTIPLE_NEXT, DB_MULTIPLE_KEY_NEXT, and DB_MULTIPLE_RECNO_NEXT macros are called repeatedly, until the end of the returned records is reached. The end of the returned records is detected by the application's local pointer variable being set to NULL.

The following is an example of a routine that displays the contents of a Btree database using the bulk return interfaces.

int
rec_display(dbp)
	DB *dbp;
{
	DBC *dbcp;
	DBT key, data;
	size_t retklen, retdlen;
	char *retkey, *retdata;
	int ret, t_ret;
	void *p;

	memset(&key, 0, sizeof(key));
	memset(&data, 0, sizeof(data));

	/* Review the database in 5MB chunks. */
#define	BUFFER_LENGTH	(5 * 1024 * 1024)
	if ((data.data = malloc(BUFFER_LENGTH)) == NULL)
		return (errno);
	data.ulen = BUFFER_LENGTH;
	data.flags = DB_DBT_USERMEM;

	/* Acquire a cursor for the database. */
	if ((ret = dbp->cursor(dbp, NULL, &dbcp, 0)) != 0) {
		dbp->err(dbp, ret, "DB->cursor");
		free(data.data);
		return (ret);
	}

	for (;;) {
		/*
		 * Acquire the next set of key/data pairs.  This code does
		 * not handle single key/data pairs that won't fit in a
		 * BUFFER_LENGTH size buffer, instead returning ENOMEM to
		 * our caller.
		 */
		if ((ret = dbcp->c_get(dbcp,
		    &key, &data, DB_MULTIPLE_KEY | DB_NEXT)) != 0) {
			if (ret != DB_NOTFOUND)
				dbp->err(dbp, ret, "DBcursor->c_get");
			break;
		}

		for (DB_MULTIPLE_INIT(p, &data);;) {
			DB_MULTIPLE_KEY_NEXT(p,
			    &data, retkey, retklen, retdata, retdlen);
			if (p == NULL)
				break;
			printf("key: %.*s, data: %.*s\n",
			    (int)retklen, retkey, (int)retdlen, retdata);
		}
	}

	if ((t_ret = dbcp->c_close(dbcp)) != 0) {
		dbp->err(dbp, ret, "DBcursor->close");
		if (ret == 0)
			ret = t_ret;
	}

	free(data.data);

	return (ret);
}

Berkeley DB Reference Guide:Access Methods

Retrieving records in bulk

Berkeley DB Reference Guide:
Access Methods